Merge pull request #359 from AikidoSec/feature/new-package-list-pypi

Add minimum package age check for pypi
This commit is contained in:
bitterpanda 2026-03-30 11:18:36 -07:00 committed by GitHub
commit 5e63a83238
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 624 additions and 298 deletions

View file

@ -111,17 +111,20 @@ safe-chain --version
The Aikido Safe Chain works by running a lightweight proxy server that intercepts package downloads from the npm registry and PyPI. When you run npm, npx, yarn, pnpm, pnpx, bun, bunx, pip, pip3, uv, poetry or pipx commands, all package downloads are routed through this local proxy, which verifies packages in real-time against **[Aikido Intel - Open Sources Threat Intelligence](https://intel.aikido.dev/?tab=malware)**. If malware is detected in any package (including deep dependencies), the proxy blocks the download before the malicious code reaches your machine. The Aikido Safe Chain works by running a lightweight proxy server that intercepts package downloads from the npm registry and PyPI. When you run npm, npx, yarn, pnpm, pnpx, bun, bunx, pip, pip3, uv, poetry or pipx commands, all package downloads are routed through this local proxy, which verifies packages in real-time against **[Aikido Intel - Open Sources Threat Intelligence](https://intel.aikido.dev/?tab=malware)**. If malware is detected in any package (including deep dependencies), the proxy blocks the download before the malicious code reaches your machine.
### Minimum package age (npm only) ### Minimum package age
For npm packages, Safe Chain applies minimum package age checks in two ways: Safe Chain applies minimum package age checks to supported ecosystems.
- During normal package resolution, Safe Chain suppresses versions that are newer than the configured minimum age from the package metadata returned by the registry. Current enforcement differs by ecosystem:
- For direct package download requests that bypass that metadata flow, Safe Chain can block the request itself using a cached list of newly released packages.
- npm-based package managers:
- during normal package resolution, Safe Chain suppresses versions that are newer than the configured minimum age from the package metadata returned by the registry
- for direct package download requests that bypass that metadata flow, Safe Chain can block the request itself using a cached list of newly released packages
- Python package managers:
- Safe Chain blocks direct package download requests using a cached list of newly released packages
By default, the minimum package age is 48 hours. This provides an additional security layer during the critical period when newly published packages are most vulnerable to containing undetected threats. You can configure this threshold or bypass this protection entirely - see the [Minimum Package Age Configuration](#minimum-package-age) section below. By default, the minimum package age is 48 hours. This provides an additional security layer during the critical period when newly published packages are most vulnerable to containing undetected threats. You can configure this threshold or bypass this protection entirely - see the [Minimum Package Age Configuration](#minimum-package-age) section below.
⚠️ This feature **only applies to npm-based package managers** (npm, npx, yarn, pnpm, pnpx, bun, bunx) and does not apply to Python package managers (uv, pip, pip3, poetry, pipx).
### Shell Integration ### Shell Integration
The Aikido Safe Chain integrates with your shell to provide a seamless experience when using npm, npx, yarn, pnpm, pnpx, bun, bunx, and Python package managers (pip, uv, poetry, pipx). It sets up aliases for these commands so that they are wrapped by the Aikido Safe Chain commands, which manage the proxy server before executing the original commands. We currently support: The Aikido Safe Chain integrates with your shell to provide a seamless experience when using npm, npx, yarn, pnpm, pnpx, bun, bunx, and Python package managers (pip, uv, poetry, pipx). It sets up aliases for these commands so that they are wrapped by the Aikido Safe Chain commands, which manage the proxy server before executing the original commands. We currently support:
@ -188,13 +191,15 @@ You can set the logging level through multiple sources (in order of priority):
## Minimum Package Age ## Minimum Package Age
You can configure how long packages must exist before Safe Chain allows their installation. By default, packages must be at least 48 hours old before they can be installed through npm-based package managers. You can configure how long packages must exist before Safe Chain allows their installation. By default, packages must be at least 48 hours old before they can be installed.
For npm-based package managers, this check currently has two enforcement modes: For npm-based package managers, this check currently has two enforcement modes:
- Safe Chain suppresses too-young versions from package metadata during normal dependency resolution. - Safe Chain suppresses too-young versions from package metadata during normal dependency resolution.
- Safe Chain blocks direct package download requests when they are matched against the cached newly released packages list. - Safe Chain blocks direct package download requests when they are matched against the cached newly released packages list.
For Python package managers, Safe Chain currently enforces minimum package age by blocking direct package download requests when they are matched against the cached newly released packages list.
### Configuration Options ### Configuration Options
You can set the minimum package age through multiple sources (in order of priority): You can set the minimum package age through multiple sources (in order of priority):
@ -225,13 +230,16 @@ You can set the minimum package age through multiple sources (in order of priori
Exclude trusted packages from minimum age filtering via environment variable or config file (both are merged). Use `@scope/*` to trust all packages from an organization: Exclude trusted packages from minimum age filtering via environment variable or config file (both are merged). Use `@scope/*` to trust all packages from an organization:
```shell ```shell
export SAFE_CHAIN_NPM_MINIMUM_PACKAGE_AGE_EXCLUSIONS="@aikidosec/*" export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="@aikidosec/*"
``` ```
```json ```json
{ {
"npm": { "npm": {
"minimumPackageAgeExclusions": ["@aikidosec/*"] "minimumPackageAgeExclusions": ["@aikidosec/*"]
},
"pip": {
"minimumPackageAgeExclusions": ["requests"]
} }
} }
``` ```

View file

@ -129,18 +129,21 @@ export function getPipCustomRegistries() {
} }
/** /**
* Gets the minimum package age exclusions from the config file * Gets the minimum package age exclusions from the config file for the current ecosystem
* @returns {string[]} * @returns {string[]}
*/ */
export function getNpmMinimumPackageAgeExclusions() { export function getMinimumPackageAgeExclusions() {
const config = readConfigFile(); const config = readConfigFile();
const ecosystem = getEcoSystem();
const registryConfig = ecosystem === "py" ? config.pip : config.npm;
if (!config || !config.npm) { if (!config || !registryConfig) {
return []; return [];
} }
const npmConfig = /** @type {SafeChainRegistryConfiguration} */ (config.npm); const typedRegistryConfig =
const exclusions = npmConfig.minimumPackageAgeExclusions; /** @type {SafeChainRegistryConfiguration} */ (registryConfig);
const exclusions = typedRegistryConfig.minimumPackageAgeExclusions;
if (!Array.isArray(exclusions)) { if (!Array.isArray(exclusions)) {
return []; return [];

View file

@ -41,6 +41,7 @@ export function getLoggingLevel() {
* Example: "react,@aikidosec/safe-chain,lodash" * Example: "react,@aikidosec/safe-chain,lodash"
* @returns {string | undefined} * @returns {string | undefined}
*/ */
export function getNpmMinimumPackageAgeExclusions() { export function getMinimumPackageAgeExclusions() {
return process.env.SAFE_CHAIN_NPM_MINIMUM_PACKAGE_AGE_EXCLUSIONS; return process.env.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS ||
process.env.SAFE_CHAIN_NPM_MINIMUM_PACKAGE_AGE_EXCLUSIONS;
} }

View file

@ -188,11 +188,11 @@ function parseExclusionsFromEnv(envValue) {
* Gets the minimum package age exclusions from both environment variable and config file (merged) * Gets the minimum package age exclusions from both environment variable and config file (merged)
* @returns {string[]} * @returns {string[]}
*/ */
export function getNpmMinimumPackageAgeExclusions() { export function getMinimumPackageAgeExclusions() {
const envExclusions = parseExclusionsFromEnv( const envExclusions = parseExclusionsFromEnv(
environmentVariables.getNpmMinimumPackageAgeExclusions() environmentVariables.getMinimumPackageAgeExclusions()
); );
const configExclusions = configFile.getNpmMinimumPackageAgeExclusions(); const configExclusions = configFile.getMinimumPackageAgeExclusions();
// Merge both sources and remove duplicates // Merge both sources and remove duplicates
const allExclusions = [...envExclusions, ...configExclusions]; const allExclusions = [...envExclusions, ...configExclusions];

View file

@ -14,7 +14,10 @@ mock.module("fs", {
const { const {
getNpmCustomRegistries, getNpmCustomRegistries,
getPipCustomRegistries, getPipCustomRegistries,
getNpmMinimumPackageAgeExclusions, getMinimumPackageAgeExclusions,
setEcoSystem,
ECOSYSTEM_JS,
ECOSYSTEM_PY,
getLoggingLevel, getLoggingLevel,
LOGGING_SILENT, LOGGING_SILENT,
LOGGING_NORMAL, LOGGING_NORMAL,
@ -367,13 +370,18 @@ describe("getLoggingLevel", () => {
}); });
}); });
describe("getNpmMinimumPackageAgeExclusions", () => { describe("getMinimumPackageAgeExclusions", () => {
let originalEnv; let originalEnv;
const envVarName = "SAFE_CHAIN_NPM_MINIMUM_PACKAGE_AGE_EXCLUSIONS"; let originalLegacyEnv;
const envVarName = "SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS";
const legacyEnvVarName = "SAFE_CHAIN_NPM_MINIMUM_PACKAGE_AGE_EXCLUSIONS";
beforeEach(() => { beforeEach(() => {
originalEnv = process.env[envVarName]; originalEnv = process.env[envVarName];
originalLegacyEnv = process.env[legacyEnvVarName];
delete process.env[envVarName]; delete process.env[envVarName];
delete process.env[legacyEnvVarName];
setEcoSystem(ECOSYSTEM_JS);
}); });
afterEach(() => { afterEach(() => {
@ -382,13 +390,18 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
} else { } else {
delete process.env[envVarName]; delete process.env[envVarName];
} }
if (originalLegacyEnv !== undefined) {
process.env[legacyEnvVarName] = originalLegacyEnv;
} else {
delete process.env[legacyEnvVarName];
}
configFileContent = undefined; configFileContent = undefined;
}); });
it("should return empty array when no exclusions configured", () => { it("should return empty array when no exclusions configured", () => {
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, []); assert.deepStrictEqual(exclusions, []);
}); });
@ -400,7 +413,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
}, },
}); });
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["react", "@aikidosec/safe-chain"]); assert.deepStrictEqual(exclusions, ["react", "@aikidosec/safe-chain"]);
}); });
@ -409,7 +422,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
process.env[envVarName] = "lodash,express,@types/node"; process.env[envVarName] = "lodash,express,@types/node";
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "express", "@types/node"]); assert.deepStrictEqual(exclusions, ["lodash", "express", "@types/node"]);
}); });
@ -422,7 +435,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
}, },
}); });
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "react"]); assert.deepStrictEqual(exclusions, ["lodash", "react"]);
}); });
@ -435,7 +448,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
}, },
}); });
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "react", "express"]); assert.deepStrictEqual(exclusions, ["lodash", "react", "express"]);
}); });
@ -444,7 +457,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
process.env[envVarName] = " lodash , react "; process.env[envVarName] = " lodash , react ";
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "react"]); assert.deepStrictEqual(exclusions, ["lodash", "react"]);
}); });
@ -456,7 +469,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
}, },
}); });
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["@babel/core", "@types/react"]); assert.deepStrictEqual(exclusions, ["@babel/core", "@types/react"]);
}); });
@ -465,7 +478,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
process.env[envVarName] = "lodash,,react,"; process.env[envVarName] = "lodash,,react,";
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "react"]); assert.deepStrictEqual(exclusions, ["lodash", "react"]);
}); });
@ -474,7 +487,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
process.env[envVarName] = ""; process.env[envVarName] = "";
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, []); assert.deepStrictEqual(exclusions, []);
}); });
@ -483,7 +496,7 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
process.env[envVarName] = " , , "; process.env[envVarName] = " , , ";
configFileContent = undefined; configFileContent = undefined;
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, []); assert.deepStrictEqual(exclusions, []);
}); });
@ -495,8 +508,29 @@ describe("getNpmMinimumPackageAgeExclusions", () => {
}, },
}); });
const exclusions = getNpmMinimumPackageAgeExclusions(); const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["react", "lodash"]); assert.deepStrictEqual(exclusions, ["react", "lodash"]);
}); });
it("should fall back to the legacy npm environment variable", () => {
process.env[legacyEnvVarName] = "lodash,react";
const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["lodash", "react"]);
});
it("should read exclusions from the python config when the current ecosystem is py", () => {
setEcoSystem(ECOSYSTEM_PY);
configFileContent = JSON.stringify({
pip: {
minimumPackageAgeExclusions: ["requests", "urllib3"],
},
});
const exclusions = getMinimumPackageAgeExclusions();
assert.deepStrictEqual(exclusions, ["requests", "urllib3"]);
});
}); });

View file

@ -4,7 +4,7 @@ import {
getEcoSystem, getEcoSystem,
} from "../../config/settings.js"; } from "../../config/settings.js";
import { npmInterceptorForUrl } from "./npm/npmInterceptor.js"; import { npmInterceptorForUrl } from "./npm/npmInterceptor.js";
import { pipInterceptorForUrl } from "./pipInterceptor.js"; import { pipInterceptorForUrl } from "./pip/pipInterceptor.js";
/** /**
* @param {string} url * @param {string} url

View file

@ -0,0 +1,33 @@
import { getMinimumPackageAgeExclusions, getEcoSystem } from "../../config/settings.js";
import { getEquivalentPackageNames } from "../../scanning/packageNameVariants.js";
/**
* Checks if a package name matches an exclusion pattern.
* Supports trailing wildcard (*) for prefix matching.
* @param {string} packageName
* @param {string} pattern
* @returns {boolean}
*/
export function matchesExclusionPattern(packageName, pattern) {
if (pattern.endsWith("/*")) {
return packageName.startsWith(pattern.slice(0, -1));
}
return packageName === pattern;
}
/**
* @param {string | undefined} packageName
* @returns {boolean}
*/
export function isExcludedFromMinimumPackageAge(packageName) {
if (!packageName) {
return false;
}
const exclusions = getMinimumPackageAgeExclusions();
const candidateNames = getEquivalentPackageNames(packageName, getEcoSystem());
return exclusions.some((pattern) =>
candidateNames.some((name) => matchesExclusionPattern(name, pattern))
);
}

View file

@ -196,17 +196,3 @@ export function getPackageNameFromMetadataResponse(body, headers) {
return undefined; return undefined;
} }
} }
/**
* Checks if a package name matches an exclusion pattern.
* Supports trailing wildcard (*) for prefix matching.
* @param {string} packageName
* @param {string} pattern
* @returns {boolean}
*/
export function matchesExclusionPattern(packageName, pattern) {
if (pattern.endsWith("/*")) {
return packageName.startsWith(pattern.slice(0, -1));
}
return packageName === pattern;
}

View file

@ -1,6 +1,5 @@
import { import {
getNpmCustomRegistries, getNpmCustomRegistries,
getNpmMinimumPackageAgeExclusions,
skipMinimumPackageAge, skipMinimumPackageAge,
} from "../../../config/settings.js"; } from "../../../config/settings.js";
import { isMalwarePackage } from "../../../scanning/audit/index.js"; import { isMalwarePackage } from "../../../scanning/audit/index.js";
@ -8,12 +7,14 @@ import { interceptRequests } from "../interceptorBuilder.js";
import { import {
getPackageNameFromMetadataResponse, getPackageNameFromMetadataResponse,
isPackageInfoUrl, isPackageInfoUrl,
matchesExclusionPattern,
modifyNpmInfoRequestHeaders, modifyNpmInfoRequestHeaders,
modifyNpmInfoResponse, modifyNpmInfoResponse,
} from "./modifyNpmInfo.js"; } from "./modifyNpmInfo.js";
import { parseNpmPackageUrl } from "./parseNpmPackageUrl.js"; import { parseNpmPackageUrl } from "./parseNpmPackageUrl.js";
import { openNewPackagesDatabase } from "../../../scanning/newPackagesListCache.js"; import { openNewPackagesDatabase } from "../../../scanning/newPackagesListCache.js";
import {
isExcludedFromMinimumPackageAge,
} from "../minimumPackageAgeExclusions.js";
const knownJsRegistries = [ const knownJsRegistries = [
"registry.npmjs.org", "registry.npmjs.org",
@ -81,17 +82,6 @@ function buildNpmInterceptor(registry) {
}); });
} }
/**
* @param {string} packageName
* @returns {boolean}
*/
function isExcludedFromMinimumPackageAge(packageName) {
const exclusions = getNpmMinimumPackageAgeExclusions();
return exclusions.some((pattern) =>
matchesExclusionPattern(packageName, pattern)
);
}
/** /**
* @param {Buffer} body * @param {Buffer} body
* @param {NodeJS.Dict<string | string[]> | undefined} headers * @param {NodeJS.Dict<string | string[]> | undefined} headers

View file

@ -14,7 +14,7 @@ describe("npmInterceptor minimum package age", async () => {
getMinimumPackageAgeHours: () => minimumPackageAgeSettings, getMinimumPackageAgeHours: () => minimumPackageAgeSettings,
skipMinimumPackageAge: () => skipMinimumPackageAgeSetting, skipMinimumPackageAge: () => skipMinimumPackageAgeSetting,
getNpmCustomRegistries: () => [], getNpmCustomRegistries: () => [],
getNpmMinimumPackageAgeExclusions: () => minimumPackageAgeExclusionsSetting, getMinimumPackageAgeExclusions: () => minimumPackageAgeExclusionsSetting,
getEcoSystem: () => "js", getEcoSystem: () => "js",
}, },
}); });

View file

@ -28,7 +28,7 @@ mock.module("../../../config/settings.js", {
setEcoSystem: () => {}, setEcoSystem: () => {},
getMinimumPackageAgeHours: () => 24, getMinimumPackageAgeHours: () => 24,
getNpmCustomRegistries: () => customRegistries, getNpmCustomRegistries: () => customRegistries,
getNpmMinimumPackageAgeExclusions: () => [], getMinimumPackageAgeExclusions: () => [],
skipMinimumPackageAge: () => skipMinimumPackageAgeSetting, skipMinimumPackageAge: () => skipMinimumPackageAgeSetting,
}, },
}); });

View file

@ -0,0 +1,101 @@
/**
* Parse Python package artifact URLs from PyPI-style registries.
* Examples:
* - Wheel: https://files.pythonhosted.org/packages/.../requests-2.28.1-py3-none-any.whl
* - Wheel metadata: https://files.pythonhosted.org/packages/.../requests-2.28.1-py3-none-any.whl.metadata
* - Sdist: https://files.pythonhosted.org/packages/.../requests-2.28.1.tar.gz
*
* @param {string} url
* @param {string} registry
* @returns {{packageName: string | undefined, version: string | undefined}}
*/
export function parsePipPackageFromUrl(url, registry) {
if (!registry || typeof url !== "string") {
return { packageName: undefined, version: undefined };
}
let urlObj;
try {
urlObj = new URL(url);
} catch {
return { packageName: undefined, version: undefined };
}
const lastSegment = urlObj.pathname.split("/").filter(Boolean).pop();
if (!lastSegment) {
return { packageName: undefined, version: undefined };
}
const filename = decodeURIComponent(lastSegment);
const wheelExtRe = /\.whl(?:\.metadata)?$/;
if (wheelExtRe.test(filename)) {
return parseWheelFilename(filename, wheelExtRe);
}
const sdistExtWithMetadataRe = /\.(tar\.gz|zip|tar\.bz2|tar\.xz)(\.metadata)?$/i;
if (!sdistExtWithMetadataRe.test(filename)) {
return { packageName: undefined, version: undefined };
}
return parseSdistFilename(filename, sdistExtWithMetadataRe);
}
/**
* Parse wheel filenames and Poetry preflight metadata.
* Examples:
* - foo_bar-2.0.0-py3-none-any.whl
* - foo_bar-2.0.0-py3-none-any.whl.metadata
*
* @param {string} filename
* @param {RegExp} wheelExtRe
* @returns {{packageName: string | undefined, version: string | undefined}}
*/
function parseWheelFilename(filename, wheelExtRe) {
const base = filename.replace(wheelExtRe, "");
const firstDash = base.indexOf("-");
if (firstDash <= 0) {
return { packageName: undefined, version: undefined };
}
const packageName = base.slice(0, firstDash);
const rest = base.slice(firstDash + 1);
const secondDash = rest.indexOf("-");
const version = secondDash >= 0 ? rest.slice(0, secondDash) : rest;
// "latest" is a resolver-style token, not an actual published artifact version.
if (version === "latest" || !packageName || !version) {
return { packageName: undefined, version: undefined };
}
return { packageName, version };
}
/**
* Parse source distribution filenames, with optional metadata suffix.
* Examples:
* - requests-2.28.1.tar.gz
* - requests-2.28.1.zip
* - requests-2.28.1.tar.gz.metadata
*
* @param {string} filename
* @param {RegExp} sdistExtWithMetadataRe
* @returns {{packageName: string | undefined, version: string | undefined}}
*/
function parseSdistFilename(filename, sdistExtWithMetadataRe) {
const base = filename.replace(sdistExtWithMetadataRe, "");
const lastDash = base.lastIndexOf("-");
if (lastDash <= 0 || lastDash >= base.length - 1) {
return { packageName: undefined, version: undefined };
}
const packageName = base.slice(0, lastDash);
const version = base.slice(lastDash + 1);
// "latest" is a resolver-style token, not an actual published artifact version.
if (version === "latest" || !packageName || !version) {
return { packageName: undefined, version: undefined };
}
return { packageName, version };
}

View file

@ -2,20 +2,32 @@ import { describe, it, mock } from "node:test";
import assert from "node:assert"; import assert from "node:assert";
describe("pipInterceptor custom registries", async () => { describe("pipInterceptor custom registries", async () => {
let lastPackage; let scannedPackages;
let malwareResponse = false; let malwareResponse = false;
let customRegistries = []; let customRegistries = [];
mock.module("../../config/settings.js", { mock.module("../../../config/settings.js", {
namedExports: { namedExports: {
ECOSYSTEM_PY: "py",
getEcoSystem: () => "py",
getMinimumPackageAgeExclusions: () => [],
getPipCustomRegistries: () => customRegistries, getPipCustomRegistries: () => customRegistries,
skipMinimumPackageAge: () => false,
}, },
}); });
mock.module("../../scanning/audit/index.js", { mock.module("../../../scanning/newPackagesListCache.js", {
namedExports: {
openNewPackagesDatabase: async () => ({
isNewlyReleasedPackage: () => false,
}),
},
});
mock.module("../../../scanning/audit/index.js", {
namedExports: { namedExports: {
isMalwarePackage: async (packageName, version) => { isMalwarePackage: async (packageName, version) => {
lastPackage = { packageName, version }; scannedPackages.push({ packageName, version });
return malwareResponse; return malwareResponse;
}, },
}, },
@ -30,42 +42,45 @@ describe("pipInterceptor custom registries", async () => {
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok( assert.ok(interceptor);
interceptor,
"Interceptor should be created for custom registry"
);
}); });
it("should parse package from custom registry URL", async () => { it("should parse package from custom registry URL", async () => {
scannedPackages = [];
customRegistries = ["my-custom-registry.example.com"]; customRegistries = ["my-custom-registry.example.com"];
const url = const url =
"https://my-custom-registry.example.com/packages/xx/yy/foobar-1.2.3.tar.gz"; "https://my-custom-registry.example.com/packages/xx/yy/foobar-1.2.3.tar.gz";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok(interceptor, "Interceptor should be created"); assert.ok(interceptor);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, { assert.ok(
packageName: "foobar", scannedPackages.some(
version: "1.2.3", ({ packageName, version }) =>
}); packageName === "foobar" && version === "1.2.3"
)
);
}); });
it("should parse wheel package from custom registry URL", async () => { it("should parse wheel package from custom registry URL", async () => {
scannedPackages = [];
customRegistries = ["private-pypi.internal.com"]; customRegistries = ["private-pypi.internal.com"];
const url = const url =
"https://private-pypi.internal.com/packages/foo_bar-2.0.0-py3-none-any.whl"; "https://private-pypi.internal.com/packages/foo_bar-2.0.0-py3-none-any.whl";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok(interceptor, "Interceptor should be created"); assert.ok(interceptor);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, { assert.ok(
packageName: "foo-bar", scannedPackages.some(
version: "2.0.0", ({ packageName, version }) =>
}); packageName === "foo-bar" && version === "2.0.0"
)
);
}); });
it("should handle multiple custom registries", async () => { it("should handle multiple custom registries", async () => {
@ -82,14 +97,12 @@ describe("pipInterceptor custom registries", async () => {
const interceptor1 = pipInterceptorForUrl(url1); const interceptor1 = pipInterceptorForUrl(url1);
const interceptor2 = pipInterceptorForUrl(url2); const interceptor2 = pipInterceptorForUrl(url2);
assert.ok(interceptor1, "Interceptor should be created for first registry"); assert.ok(interceptor1);
assert.ok( assert.ok(interceptor2);
interceptor2,
"Interceptor should be created for second registry"
);
}); });
it("should block malicious package from custom registry", async () => { it("should block malicious package from custom registry", async () => {
scannedPackages = [];
customRegistries = ["my-custom-registry.example.com"]; customRegistries = ["my-custom-registry.example.com"];
malwareResponse = true; malwareResponse = true;
@ -97,26 +110,19 @@ describe("pipInterceptor custom registries", async () => {
"https://my-custom-registry.example.com/packages/malicious_package-1.0.0.tar.gz"; "https://my-custom-registry.example.com/packages/malicious_package-1.0.0.tar.gz";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok(interceptor, "Interceptor should be created"); assert.ok(interceptor);
const result = await interceptor.handleRequest(url); const result = await interceptor.handleRequest(url);
assert.ok(result.blockResponse, "Should contain a blockResponse"); assert.ok(result.blockResponse);
assert.equal( assert.equal(result.blockResponse.statusCode, 403);
result.blockResponse.statusCode, assert.equal(result.blockResponse.message, "Forbidden - blocked by safe-chain");
403,
"Block response should have status code 403"
);
assert.equal(
result.blockResponse.message,
"Forbidden - blocked by safe-chain",
"Block response should have correct status message"
);
malwareResponse = false; malwareResponse = false;
}); });
it("should still work with known registries when custom registries are set", async () => { it("should still work with known registries when custom registries are set", async () => {
scannedPackages = [];
customRegistries = ["my-custom-registry.example.com"]; customRegistries = ["my-custom-registry.example.com"];
const url = const url =
@ -124,17 +130,16 @@ describe("pipInterceptor custom registries", async () => {
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok( assert.ok(interceptor);
interceptor,
"Interceptor should be created for known registry even with custom registries set"
);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, { assert.ok(
packageName: "foobar", scannedPackages.some(
version: "1.2.3", ({ packageName, version }) =>
}); packageName === "foobar" && version === "1.2.3"
)
);
}); });
it("should not create interceptor for unknown registry when custom registries are set", () => { it("should not create interceptor for unknown registry when custom registries are set", () => {
@ -143,11 +148,7 @@ describe("pipInterceptor custom registries", async () => {
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.equal( assert.equal(interceptor, undefined);
interceptor,
undefined,
"Interceptor should be undefined for unknown registry"
);
}); });
it("should handle empty custom registries array", () => { it("should handle empty custom registries array", () => {
@ -157,43 +158,44 @@ describe("pipInterceptor custom registries", async () => {
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.equal( assert.equal(interceptor, undefined);
interceptor,
undefined,
"Interceptor should be undefined when no custom registries are configured"
);
}); });
it("should parse .whl.metadata from custom registry", async () => { it("should parse .whl.metadata from custom registry", async () => {
scannedPackages = [];
customRegistries = ["private-pypi.internal.com"]; customRegistries = ["private-pypi.internal.com"];
const url = const url =
"https://private-pypi.internal.com/packages/foo_bar-2.0.0-py3-none-any.whl.metadata"; "https://private-pypi.internal.com/packages/foo_bar-2.0.0-py3-none-any.whl.metadata";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok(interceptor, "Interceptor should be created"); assert.ok(interceptor);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, { assert.ok(
packageName: "foo-bar", scannedPackages.some(
version: "2.0.0", ({ packageName, version }) =>
}); packageName === "foo-bar" && version === "2.0.0"
)
);
}); });
it("should parse .tar.gz.metadata from custom registry", async () => { it("should parse .tar.gz.metadata from custom registry", async () => {
scannedPackages = [];
customRegistries = ["private-pypi.internal.com"]; customRegistries = ["private-pypi.internal.com"];
const url = const url =
"https://private-pypi.internal.com/packages/foo_bar-2.0.0.tar.gz.metadata"; "https://private-pypi.internal.com/packages/foo_bar-2.0.0.tar.gz.metadata";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok(interceptor, "Interceptor should be created"); assert.ok(interceptor);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, { assert.ok(
packageName: "foo-bar", scannedPackages.some(
version: "2.0.0", ({ packageName, version }) =>
packageName === "foo-bar" && version === "2.0.0"
)
);
}); });
}); });
});

View file

@ -0,0 +1,96 @@
import {
ECOSYSTEM_PY,
getPipCustomRegistries,
skipMinimumPackageAge,
} from "../../../config/settings.js";
import { isMalwarePackage } from "../../../scanning/audit/index.js";
import { getEquivalentPackageNames } from "../../../scanning/packageNameVariants.js";
import { openNewPackagesDatabase } from "../../../scanning/newPackagesListCache.js";
import { interceptRequests } from "../interceptorBuilder.js";
import { isExcludedFromMinimumPackageAge } from "../minimumPackageAgeExclusions.js";
import { parsePipPackageFromUrl } from "./parsePipPackageUrl.js";
const knownPipRegistries = [
"files.pythonhosted.org",
"pypi.org",
"pypi.python.org",
"pythonhosted.org",
];
/**
* @param {string} url
* @returns {import("../interceptorBuilder.js").Interceptor | undefined}
*/
export function pipInterceptorForUrl(url) {
const customRegistries = getPipCustomRegistries();
const registries = [...knownPipRegistries, ...customRegistries];
const registry = registries.find((reg) => url.includes(reg));
if (registry) {
return buildPipInterceptor(registry);
}
return undefined;
}
/**
* @param {string} registry
* @returns {import("../interceptorBuilder.js").Interceptor | undefined}
*/
function buildPipInterceptor(registry) {
return interceptRequests(createPipRequestHandler(registry));
}
/**
* @param {string} registry
* @returns {(reqContext: import("../interceptorBuilder.js").RequestInterceptionContext) => Promise<void>}
*/
function createPipRequestHandler(registry) {
return async (reqContext) => {
const { packageName, version } = parsePipPackageFromUrl(
reqContext.targetUrl,
registry
);
if (!packageName) {
return;
}
const equivalentPackageNames = getEquivalentPackageNames(
packageName,
ECOSYSTEM_PY
);
let isMalicious = false;
for (const equivalentPackageName of equivalentPackageNames) {
if (await isMalwarePackage(equivalentPackageName, version)) {
isMalicious = true;
break;
}
}
if (isMalicious) {
reqContext.blockMalware(packageName, version);
return;
}
if (
version &&
!skipMinimumPackageAge() &&
!isExcludedFromMinimumPackageAge(packageName)
) {
const newPackagesDatabase = await openNewPackagesDatabase();
const isNewlyReleased = newPackagesDatabase.isNewlyReleasedPackage(
packageName,
version
);
if (isNewlyReleased) {
reqContext.blockMinimumAgeRequest(
packageName,
version,
`Forbidden - blocked by safe-chain direct download minimum package age (${packageName}@${version})`
);
}
}
};
}

View file

@ -0,0 +1,103 @@
import { describe, it, mock } from "node:test";
import assert from "node:assert";
describe("pipInterceptor minimum package age", async () => {
let skipMinimumPackageAgeSetting = false;
let newlyReleasedPackageResponse = false;
let minimumPackageAgeExclusionsSetting = [];
mock.module("../../../scanning/audit/index.js", {
namedExports: {
isMalwarePackage: async () => false,
},
});
mock.module("../../../scanning/newPackagesListCache.js", {
namedExports: {
openNewPackagesDatabase: async () => ({
isNewlyReleasedPackage: (packageName, version) => {
return newlyReleasedPackageResponse &&
(packageName === "foo-bar" ||
packageName === "foo_bar" ||
packageName === "foo.bar") &&
version === "2.0.0";
},
}),
},
});
mock.module("../../../config/settings.js", {
namedExports: {
ECOSYSTEM_PY: "py",
getEcoSystem: () => "py",
getMinimumPackageAgeExclusions: () => minimumPackageAgeExclusionsSetting,
getPipCustomRegistries: () => [],
skipMinimumPackageAge: () => skipMinimumPackageAgeSetting,
},
});
const { pipInterceptorForUrl } = await import("./pipInterceptor.js");
it("should block newly released package downloads", async () => {
const url =
"https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0-py3-none-any.whl";
newlyReleasedPackageResponse = true;
const interceptor = pipInterceptorForUrl(url);
const result = await interceptor.handleRequest(url);
assert.ok(result.blockResponse);
assert.equal(result.blockResponse.statusCode, 403);
assert.equal(
result.blockResponse.message,
"Forbidden - blocked by safe-chain direct download minimum package age (foo_bar@2.0.0)"
);
newlyReleasedPackageResponse = false;
});
it("should not block newly released package downloads when skipMinimumPackageAge is enabled", async () => {
const url =
"https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0-py3-none-any.whl";
newlyReleasedPackageResponse = true;
skipMinimumPackageAgeSetting = true;
const interceptor = pipInterceptorForUrl(url);
const result = await interceptor.handleRequest(url);
assert.equal(result.blockResponse, undefined);
skipMinimumPackageAgeSetting = false;
newlyReleasedPackageResponse = false;
});
it("should not block newly released package downloads when the package is excluded", async () => {
const url =
"https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0-py3-none-any.whl";
newlyReleasedPackageResponse = true;
minimumPackageAgeExclusionsSetting = ["foo-bar"];
const interceptor = pipInterceptorForUrl(url);
const result = await interceptor.handleRequest(url);
assert.equal(result.blockResponse, undefined);
minimumPackageAgeExclusionsSetting = [];
newlyReleasedPackageResponse = false;
});
it("should not block newly released package downloads when a dot-name package matches a hyphen exclusion", async () => {
const url =
"https://files.pythonhosted.org/packages/xx/yy/foo.bar-2.0.0.tar.gz";
newlyReleasedPackageResponse = true;
minimumPackageAgeExclusionsSetting = ["foo-bar"];
const interceptor = pipInterceptorForUrl(url);
const result = await interceptor.handleRequest(url);
assert.equal(result.blockResponse, undefined);
minimumPackageAgeExclusionsSetting = [];
newlyReleasedPackageResponse = false;
});
});

View file

@ -2,22 +2,39 @@ import { describe, it, mock } from "node:test";
import assert from "node:assert"; import assert from "node:assert";
describe("pipInterceptor", async () => { describe("pipInterceptor", async () => {
let lastPackage; let scannedPackages;
let malwareResponse = false; let malwareResponse = false;
mock.module("../../scanning/audit/index.js", { mock.module("../../../scanning/audit/index.js", {
namedExports: { namedExports: {
isMalwarePackage: async (packageName, version) => { isMalwarePackage: async (packageName, version) => {
lastPackage = { packageName, version }; scannedPackages.push({ packageName, version });
return malwareResponse; return malwareResponse;
}, },
}, },
}); });
mock.module("../../../scanning/newPackagesListCache.js", {
namedExports: {
openNewPackagesDatabase: async () => ({
isNewlyReleasedPackage: () => false,
}),
},
});
mock.module("../../../config/settings.js", {
namedExports: {
ECOSYSTEM_PY: "py",
getEcoSystem: () => "py",
getMinimumPackageAgeExclusions: () => [],
getPipCustomRegistries: () => [],
skipMinimumPackageAge: () => false,
},
});
const { pipInterceptorForUrl } = await import("./pipInterceptor.js"); const { pipInterceptorForUrl } = await import("./pipInterceptor.js");
const parserCases = [ const parserCases = [
// Valid pip URLs
{ {
url: "https://files.pythonhosted.org/packages/xx/yy/foobar-1.2.3.tar.gz", url: "https://files.pythonhosted.org/packages/xx/yy/foobar-1.2.3.tar.gz",
expected: { packageName: "foobar", version: "1.2.3" }, expected: { packageName: "foobar", version: "1.2.3" },
@ -35,7 +52,6 @@ describe("pipInterceptor", async () => {
expected: { packageName: "foo-bar", version: "2.0.0" }, expected: { packageName: "foo-bar", version: "2.0.0" },
}, },
{ {
// Poetry preflight metadata alongside wheel (.whl.metadata)
url: "https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0-py3-none-any.whl.metadata", url: "https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0-py3-none-any.whl.metadata",
expected: { packageName: "foo-bar", version: "2.0.0" }, expected: { packageName: "foo-bar", version: "2.0.0" },
}, },
@ -52,7 +68,6 @@ describe("pipInterceptor", async () => {
expected: { packageName: "foo-bar", version: "2.0.0b1" }, expected: { packageName: "foo-bar", version: "2.0.0b1" },
}, },
{ {
// sdist with metadata sidecar (.tar.gz.metadata)
url: "https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0.tar.gz.metadata", url: "https://files.pythonhosted.org/packages/xx/yy/foo_bar-2.0.0.tar.gz.metadata",
expected: { packageName: "foo-bar", version: "2.0.0" }, expected: { packageName: "foo-bar", version: "2.0.0" },
}, },
@ -76,7 +91,6 @@ describe("pipInterceptor", async () => {
url: "https://pypi.org/packages/source/f/foo_bar/foo_bar-2.0.0-cp38-cp38-manylinux1_x86_64.whl", url: "https://pypi.org/packages/source/f/foo_bar/foo_bar-2.0.0-cp38-cp38-manylinux1_x86_64.whl",
expected: { packageName: "foo-bar", version: "2.0.0" }, expected: { packageName: "foo-bar", version: "2.0.0" },
}, },
// Invalid pip URLs
{ {
url: "https://pypi.org/simple/", url: "https://pypi.org/simple/",
expected: { packageName: undefined, version: undefined }, expected: { packageName: undefined, version: undefined },
@ -97,49 +111,49 @@ describe("pipInterceptor", async () => {
parserCases.forEach(({ url, expected }, index) => { parserCases.forEach(({ url, expected }, index) => {
it(`should parse URL ${index + 1}: ${url}`, async () => { it(`should parse URL ${index + 1}: ${url}`, async () => {
scannedPackages = [];
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.ok( assert.ok(interceptor, "Interceptor should be created for known pip registry");
interceptor,
"Interceptor should be created for known npm registry"
);
await interceptor.handleRequest(url); await interceptor.handleRequest(url);
assert.deepEqual(lastPackage, expected); if (expected.packageName === undefined) {
assert.deepEqual(scannedPackages, []);
return;
}
assert.ok(
scannedPackages.some(
({ packageName, version }) =>
packageName === expected.packageName &&
version === expected.version
)
);
}); });
}); });
it("should not create interceptor for unknown registry", () => { it("should not create interceptor for unknown registry", () => {
const url = "https://example.com/packages/xx/yy/foobar-1.2.3.tar.gz"; const url = "https://example.com/packages/xx/yy/foobar-1.2.3.tar.gz";
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
assert.equal(interceptor, undefined);
assert.equal(
interceptor,
undefined,
"Interceptor should be undefined for unknown registry"
);
}); });
it("should block malicious package", async () => { it("should block malicious package", async () => {
scannedPackages = [];
const url = const url =
"https://files.pythonhosted.org/packages/xx/yy/malicious_package-1.0.0.tar.gz"; "https://files.pythonhosted.org/packages/xx/yy/malicious_package-1.0.0.tar.gz";
malwareResponse = true; malwareResponse = true;
const interceptor = pipInterceptorForUrl(url); const interceptor = pipInterceptorForUrl(url);
const result = await interceptor.handleRequest(url); const result = await interceptor.handleRequest(url);
assert.ok(result.blockResponse, "Should contain a blockResponse"); assert.ok(result.blockResponse);
assert.equal( assert.equal(result.blockResponse.statusCode, 403);
result.blockResponse.statusCode,
403,
"Block response should have status code 403"
);
assert.equal( assert.equal(
result.blockResponse.message, result.blockResponse.message,
"Forbidden - blocked by safe-chain", "Forbidden - blocked by safe-chain"
"Block response should have correct status message"
); );
malwareResponse = false;
}); });
}); });

View file

@ -1,132 +0,0 @@
import { getPipCustomRegistries } from "../../config/settings.js";
import { isMalwarePackage } from "../../scanning/audit/index.js";
import { interceptRequests } from "./interceptorBuilder.js";
const knownPipRegistries = [
"files.pythonhosted.org",
"pypi.org",
"pypi.python.org",
"pythonhosted.org",
];
/**
* @param {string} url
* @returns {import("./interceptorBuilder.js").Interceptor | undefined}
*/
export function pipInterceptorForUrl(url) {
const customRegistries = getPipCustomRegistries();
const registries = [...knownPipRegistries, ...customRegistries];
const registry = registries.find((reg) => url.includes(reg));
if (registry) {
return buildPipInterceptor(registry);
}
return undefined;
}
/**
* @param {string} registry
* @returns {import("./interceptorBuilder.js").Interceptor | undefined}
*/
function buildPipInterceptor(registry) {
return interceptRequests(async (reqContext) => {
const { packageName, version } = parsePipPackageFromUrl(
reqContext.targetUrl,
registry
);
// Normalize underscores to hyphens for DB matching, as PyPI allows underscores in distribution names.
// Per python, packages that differ only by hyphen vs underscore are considered the same.
const hyphenName = packageName?.includes("_") ? packageName.replace(/_/g, "-") : packageName;
const isMalicious =
await isMalwarePackage(packageName, version)
|| await isMalwarePackage(hyphenName, version);
if (isMalicious) {
reqContext.blockMalware(packageName, version);
}
});
}
/**
* @param {string} url
* @param {string} registry
* @returns {{packageName: string | undefined, version: string | undefined}}
*/
function parsePipPackageFromUrl(url, registry) {
let packageName, version;
// Basic validation
if (!registry || typeof url !== "string") {
return { packageName, version };
}
// Quick sanity check on the URL + parse
let urlObj;
try {
urlObj = new URL(url);
} catch {
return { packageName, version };
}
// Get the last path segment (filename) and decode it (strip query & fragment automatically)
const lastSegment = urlObj.pathname.split("/").filter(Boolean).pop();
if (!lastSegment) {
return { packageName, version };
}
const filename = decodeURIComponent(lastSegment);
// Parse Python package downloads from PyPI/pythonhosted.org
// Example wheel: https://files.pythonhosted.org/packages/xx/yy/requests-2.28.1-py3-none-any.whl
// Example sdist: https://files.pythonhosted.org/packages/xx/yy/requests-2.28.1.tar.gz
// Wheel (.whl) and Poetry's preflight metadata (.whl.metadata)
// Examples:
// foo_bar-2.0.0-py3-none-any.whl
// foo_bar-2.0.0-py3-none-any.whl.metadata
const wheelExtRe = /\.whl(?:\.metadata)?$/;
const wheelExtMatch = filename.match(wheelExtRe);
if (wheelExtMatch) {
const base = filename.replace(wheelExtRe, "");
const firstDash = base.indexOf("-");
if (firstDash > 0) {
const dist = base.slice(0, firstDash); // may contain underscores
const rest = base.slice(firstDash + 1); // version + the rest of tags
const secondDash = rest.indexOf("-");
const rawVersion = secondDash >= 0 ? rest.slice(0, secondDash) : rest;
packageName = dist;
version = rawVersion;
// Reject "latest" as it's a placeholder, not a real version
// When version is "latest", this signals the URL doesn't contain actual version info
// Returning undefined allows the request (see registryProxy.js isAllowedUrl)
if (version === "latest" || !packageName || !version) {
return { packageName: undefined, version: undefined };
}
return { packageName, version };
}
}
// Source dist (sdist) and potential metadata sidecars (e.g., .tar.gz.metadata)
const sdistExtWithMetadataRe = /\.(tar\.gz|zip|tar\.bz2|tar\.xz)(\.metadata)?$/i;
const sdistExtMatch = filename.match(sdistExtWithMetadataRe);
if (sdistExtMatch) {
const base = filename.replace(sdistExtWithMetadataRe, "");
const lastDash = base.lastIndexOf("-");
if (lastDash > 0 && lastDash < base.length - 1) {
packageName = base.slice(0, lastDash);
version = base.slice(lastDash + 1);
// Reject "latest" as it's a placeholder, not a real version
// When version is "latest", this signals the URL doesn't contain actual version info
// Returning undefined allows the request (see registryProxy.js isAllowedUrl)
if (version === "latest" || !packageName || !version) {
return { packageName: undefined, version: undefined };
}
return { packageName, version };
}
}
// Unknown file type or invalid
return { packageName: undefined, version: undefined };
}

View file

@ -174,10 +174,18 @@ describe("newPackagesDatabase", async () => {
assert.strictEqual(db.isNewlyReleasedPackage("foo", "1.0.0"), true); assert.strictEqual(db.isNewlyReleasedPackage("foo", "1.0.0"), true);
}); });
it("returns false for all packages when ecosystem is not JS", async () => { it("supports package checks for the python ecosystem", async () => {
ecosystem = "py"; ecosystem = "py";
fetchedList = [
{
source: "pypi",
package_name: "foo",
version: "1.0.0",
released_on: hoursAgo(1),
},
];
const db = await openNewPackagesDatabase(); const db = await openNewPackagesDatabase();
assert.strictEqual(db.isNewlyReleasedPackage("foo", "1.0.0"), false); assert.strictEqual(db.isNewlyReleasedPackage("foo", "1.0.0"), true);
}); });
}); });

View file

@ -4,10 +4,11 @@ import {
ECOSYSTEM_JS, ECOSYSTEM_JS,
ECOSYSTEM_PY, ECOSYSTEM_PY,
} from "../config/settings.js"; } from "../config/settings.js";
import { getEquivalentPackageNames } from "./packageNameVariants.js";
/** /**
* @typedef {Object} NewPackagesDatabase * @typedef {Object} NewPackagesDatabase
* @property {function(string, string): boolean} isNewlyReleasedPackage * @property {function(string | undefined, string | undefined): boolean} isNewlyReleasedPackage
*/ */
/** /**
@ -33,21 +34,28 @@ function getCurrentFeedSource() {
* @returns {NewPackagesDatabase} * @returns {NewPackagesDatabase}
*/ */
export function buildNewPackagesDatabase(newPackagesList) { export function buildNewPackagesDatabase(newPackagesList) {
const ecosystem = getEcoSystem();
/** /**
* @param {string} name * @param {string | undefined} name
* @param {string} version * @param {string | undefined} version
* @returns {boolean} * @returns {boolean}
*/ */
function isNewlyReleasedPackage(name, version) { function isNewlyReleasedPackage(name, version) {
if (!name || !version) {
return false;
}
const cutOff = new Date( const cutOff = new Date(
new Date().getTime() - getMinimumPackageAgeHours() * 3600 * 1000 new Date().getTime() - getMinimumPackageAgeHours() * 3600 * 1000
); );
const expectedSource = getCurrentFeedSource(); const expectedSource = getCurrentFeedSource();
const candidateNames = getEquivalentPackageNames(name, ecosystem);
const entry = newPackagesList.find( const entry = newPackagesList.find(
(pkg) => (pkg) =>
(!pkg.source || pkg.source.toLowerCase() === expectedSource) && (!pkg.source || pkg.source.toLowerCase() === expectedSource) &&
pkg.package_name === name && candidateNames.includes(pkg.package_name) &&
pkg.version === version pkg.version === version
); );

View file

@ -50,6 +50,15 @@ describe("buildNewPackagesDatabase", () => {
assert.strictEqual(db.isNewlyReleasedPackage("not-there", "1.0.0"), false); assert.strictEqual(db.isNewlyReleasedPackage("not-there", "1.0.0"), false);
}); });
it("returns false when name or version is undefined", () => {
const db = buildNewPackagesDatabase([
{ package_name: "foo", version: "1.0.0", released_on: hoursAgo(1) },
]);
assert.strictEqual(db.isNewlyReleasedPackage(undefined, "1.0.0"), false);
assert.strictEqual(db.isNewlyReleasedPackage("foo", undefined), false);
});
it("returns false for a known package but different version", () => { it("returns false for a known package but different version", () => {
const db = buildNewPackagesDatabase([ const db = buildNewPackagesDatabase([
{ package_name: "foo", version: "2.0.0", released_on: hoursAgo(1) }, { package_name: "foo", version: "2.0.0", released_on: hoursAgo(1) },
@ -96,5 +105,54 @@ describe("buildNewPackagesDatabase", () => {
minimumPackageAgeHours = 24; // reset minimumPackageAgeHours = 24; // reset
}); });
it("matches underscore request names against hyphen feed names for python", () => {
ecosystem = "py";
const db = buildNewPackagesDatabase([
{ source: "pypi", package_name: "foo-bar", version: "1.0.0", released_on: hoursAgo(1) },
]);
assert.strictEqual(db.isNewlyReleasedPackage("foo_bar", "1.0.0"), true);
ecosystem = "js";
});
it("matches hyphen request names against underscore feed names for python", () => {
ecosystem = "py";
const db = buildNewPackagesDatabase([
{ source: "pypi", package_name: "foo_bar", version: "1.0.0", released_on: hoursAgo(1) },
]);
assert.strictEqual(db.isNewlyReleasedPackage("foo-bar", "1.0.0"), true);
ecosystem = "js";
});
it("matches dot request names against hyphen feed names for python", () => {
ecosystem = "py";
const db = buildNewPackagesDatabase([
{ source: "pypi", package_name: "foo-bar", version: "1.0.0", released_on: hoursAgo(1) },
]);
assert.strictEqual(db.isNewlyReleasedPackage("foo.bar", "1.0.0"), true);
ecosystem = "js";
});
it("matches underscore request names against dot feed names for python", () => {
ecosystem = "py";
const db = buildNewPackagesDatabase([
{ source: "pypi", package_name: "foo.bar", version: "1.0.0", released_on: hoursAgo(1) },
]);
assert.strictEqual(db.isNewlyReleasedPackage("foo_bar", "1.0.0"), true);
ecosystem = "js";
});
}); });
}); });

View file

@ -8,7 +8,6 @@ import {
getNewPackagesListVersionPath, getNewPackagesListVersionPath,
} from "../config/configFile.js"; } from "../config/configFile.js";
import { ui } from "../environment/userInteraction.js"; import { ui } from "../environment/userInteraction.js";
import { getEcoSystem, ECOSYSTEM_JS } from "../config/settings.js";
import { buildNewPackagesDatabase } from "./newPackagesDatabaseBuilder.js"; import { buildNewPackagesDatabase } from "./newPackagesDatabaseBuilder.js";
import { warnOnceAboutUnavailableDatabase } from "./newPackagesDatabaseWarnings.js"; import { warnOnceAboutUnavailableDatabase } from "./newPackagesDatabaseWarnings.js";
@ -28,11 +27,6 @@ export async function openNewPackagesDatabase() {
return cachedNewPackagesDatabase; return cachedNewPackagesDatabase;
} }
if (getEcoSystem() !== ECOSYSTEM_JS) {
cachedNewPackagesDatabase = { isNewlyReleasedPackage: () => false };
return cachedNewPackagesDatabase;
}
/** @type {import("../api/aikido.js").NewPackageEntry[]} */ /** @type {import("../api/aikido.js").NewPackageEntry[]} */
let newPackagesList; let newPackagesList;

View file

@ -0,0 +1,19 @@
import { ECOSYSTEM_PY } from "../config/settings.js";
/**
* @param {string} packageName
* @param {string} ecosystem
* @returns {string[]}
*/
export function getEquivalentPackageNames(packageName, ecosystem) {
if (ecosystem !== ECOSYSTEM_PY) {
return [packageName];
}
const pythonSeparatorPattern = /[._-]/g;
const hyphenName = packageName.replaceAll(pythonSeparatorPattern, "-");
const underscoreName = packageName.replaceAll(pythonSeparatorPattern, "_");
const dotName = packageName.replaceAll(pythonSeparatorPattern, ".");
return [...new Set([packageName, hyphenName, underscoreName, dotName])];
}