Conversation
Index data was never a hard requirement for this project. Drops the public API, BaseSource contract, both source adapters' implementations, CLI commands (index-list/index-minute/index-daily), and now-dead checkpoint-fetch helpers that only those commands used. Updates README/CLAUDE.md accordingly and cancels the corresponding compare subcommands from the roadmap.
Not a hard requirement for current usage. Drops the public API, BaseSource contract, both source adapters' implementations (including tushare's NotImplementedError placeholder), the stock-minute CLI command, and the now-unused frequency/VALID_FREQUENCIES plumbing. Updates README/CLAUDE.md accordingly.
_fetch_stock_bar_by_trading_day fetched every trading day into memory, concatenated them, then split the result back apart by date to write files — pointless round-trip once each day's bars already come out of get_bar_fn scoped to that single day. Writing directly per day also buys incremental resume for free: a day whose CSV already exists is skipped, and an error partway through no longer discards bars already written for earlier days (previously the whole batch was thrown away). Drops _write_by_date, now dead since nothing merges bars across days anymore.
TestTushareIntegration and TestRicequantIntegration had ~7 test methods duplicated line-for-line (test_get_calendar, test_get_stock_list_by_*, test_get_stock_snapshot, test_get_stock_daily_bar), differing only in which self.source they called. Moves the shared methods to tests/helpers.py::IntegrationTestMixin; both classes now inherit it and keep only their setup fixture plus the one test_get_stock_list that genuinely differs per source (ricequant checks list_date/delist_date format; tushare's get_stock_daily_bar needs an explicit trading_days kwarg when called directly, handled via the _DAILY_BAR_KWARGS override).
Docstring claimed the lookback walks back up to 60 trading days; the code has always walked back up to one year. No test pins the actual lookback distance and there's no live rqdatac access here to verify the real data-availability lag, so the code (shipped, working behavior) is treated as ground truth and the docstring is corrected to match — narrowing the window instead would be guessing at something that isn't confirmed broken.
pyproject.toml is a normal tracked file with real commit history (including version bumps); it was never gitignored. Drops the false CLAUDE.md line, and the dead .gitignore negation (!pyproject.toml.example) that referenced a file that doesn't exist and cancelled no active rule.
…daily Now that stock-minute is gone, cmd_stock_daily was the only caller — the get_bar_fn/tag parameterization existed solely to share the function between two commands. Inlining it removes that indirection and brings the command in line with cmd_stock_list's style right next to it. The resume-behavior note moves from the private function's docstring to the command's own docstring, so `hqdata stock-daily --help` now surfaces it directly.
get_stock_list had two near-identical inline loops mapping comma-separated board/exchange values through a dict while preserving unrecognized entries. Dedupes into one static helper both call.
Verified with real same-day data across 主板/科创板/创业板 (600000.SH, 688981.SH, 300750.SZ): tushare's amount*1000/(vol*close) and ricequant's total_turnover/(volume*close) both land on ~100 for every board, confirming both sources report volume/vol using a uniform 1-lot-equals-100-shares convention with no board-specific handling — matching what tushare.py and ricequant.py already do uniformly. The README's "科创板1手=200股" conflated 科创板's minimum order size rule (申报买入最低200股) with the volume *reporting* unit, which are different things. No code change needed — the bug was in the docs.
…_bar Only caller was get_stock_daily_bar; the split gave no test or reuse benefit (unlike _get_hs_connect_stocks, which is independently unit tested). Inlining removes the indirection.
get_stock_list's reverse-mapping used plain .map(), which turns an unmapped exchange/board_type into NaN. tushare.py's equivalent already falls back to the raw value via .get(x, x); ricequant.py now matches. Currently unreachable in practice — verified against real all_instruments(type="CS") data back to 2018-01-01 (predating the 2021-04 SZSE SME-board merger) that exchange is always XSHE/XSHG/BJSE and board_type is always MainBoard/GEM/KSH/BJS, never anything else. Documented as comments next to the maps so the next person doesn't have to re-derive it.
Neither field is needed going forward. Removes them from the public API contract (api.py, base.py), both source adapters (tushare.py no longer even requests industry/is_hs from stock_basic; ricequant.py drops the industry_name/is_hs assignments), compare_cli's stock-list schema and normalization, and all test fixtures/mocks/assertions that referenced either field. As a result, ricequant.py's _get_hs_connect_stocks() — whose only purpose was computing is_hs — is now dead code and removed along with its now-unused `date` import. This incidentally resolves both reliability issues found earlier in that function (ignoring the queried trade_date, and paying ~29 sequential get_stock_connect calls per get_stock_list call with no caching): the function simply no longer exists. Verified against real tushare/ricequant API calls that both sources now return the identical 8-column schema: symbol, date, name, exchange, board, curr_type, list_date, delist_date.
…ssage Add get_stock_daily_bar implementation for RicequantSource using rq.get_price with 1d frequency. Also translate the rqdatac import error message from Chinese to English and reorder method definitions for logical grouping (snapshot methods after daily bar methods) in api.py and base.py.
get_stock_daily_bar sized chunks purely from the 6000-row response limit (5900 // trading_days), so a single-day query computed a chunk of 5900 codes and hit tushare's per-request ts_code cap: 列表个数超过限制1000个. The cap is undocumented but verified empirically — 1000 codes succeed, 1001 raise. Clamp chunk_size to min(5900 // trading_days, 1000). Verified the previously-failing scenario (hqdata stock-daily over a full ~5470 symbol universe) now completes.
13 tasks
There was a problem hiding this comment.
Pull request overview
This PR simplifies hqdata’s surface area by narrowing the supported data model and public API to stock-focused functionality (calendar, stock list, stock daily bars, stock snapshot), removing minute-bar and index-related APIs, and updating tests/docs/CLI accordingly.
Changes:
- Remove index and minute-bar APIs end-to-end (BaseSource contract, adapters, public API exports, CLI commands, docs, and tests).
- Simplify stock list schema by dropping
industry/is_hsand propagate the new column contract through compare tooling and tests. - Consolidate integration test coverage via a shared
IntegrationTestMixinand adjust CLI stock-daily to write per trading day with skip/resume-friendly behavior.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tushare.py | Removes duplicated integration tests and switches to shared integration mixin; updates fixtures for reduced stock-list columns. |
| tests/test_ricequant.py | Removes duplicated integration tests and switches to shared integration mixin; updates mocks/fixtures for reduced stock-list columns. |
| tests/test_package_api.py | Updates exported-API expectations after removing index/minute public APIs. |
| tests/test_cli.py | Removes index/minute CLI coverage and adds stock-daily per-day write/skip/error-path tests; updates stock-list fixtures to match new columns. |
| tests/helpers.py | Introduces IntegrationTestMixin and removes now-obsolete index/minute column contracts/helpers. |
| README.md | Updates capability table and parameter docs after dropping index/minute APIs; clarifies volume unit normalization. |
| hqdata/sources/tushare.py | Removes stock-list industry/is_hs, drops index/minute APIs, adds comma-separated mapping helper, and updates optional-dependency ImportError message. |
| hqdata/sources/ricequant.py | Removes industry/is_hs, drops index/minute APIs, inlines daily-bar normalization, and updates optional-dependency ImportError message. |
| hqdata/sources/base.py | Shrinks the adapter contract to calendar/stock-list/stock-daily/stock-snapshot and updates empty-frame column contracts accordingly. |
| hqdata/compare_cli.py | Aligns stock-list comparison schema/normalization with removed columns and updates per-date compare command wording. |
| hqdata/cli.py | Removes index/minute commands and implements stock-daily per-trading-day fetching with skip-on-existing and error-resume messaging. |
| hqdata/api.py | Removes index/minute public APIs; keeps stock-list, stock-daily, and stock-snapshot. |
| hqdata/init.py | Removes exports for index/minute public APIs. |
| CLAUDE.md | Updates repository guidance to reflect reduced public API and adapter responsibilities. |
| .gitignore | Adjusts ignore rules (notably around pyproject.toml.example). |
| .claude/commands/add-source.md | Updates “add source” guidance to implement the current BaseSource abstract methods and parameter consistency rules. |
Suppressed comments (1)
tests/test_cli.py:365
- The CLI prints error lines with
err=True, which Click writes to stderr. This assertion only checksresult.output, so it can fail depending on Click's stderr-mixing behavior. Assert against both stdout and stderr to make the test stable.
assert_success(result)
assert "ERROR" in result.output
assert (tmp_path / "tushare" / "stock_daily" / "20260102.csv").exists()
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
16
to
22
| raise ImportError( | ||
| "tushare 未安装，hqdata不会默认安装您不一定需要的依赖。请运行：pip install hqdata[tushare]开启对tushare的支持。" | ||
| """tushare is not installed. | ||
|
|
||
| hqdata does not install dependencies you may not need by default. | ||
| Please run: pip install hqdata[tushare] to enable tushare support. | ||
| """ | ||
| ) from None |
Comment on lines
16
to
22
| raise ImportError( | ||
| "rqdatac 未安装，hqdata不会默认安装您不一定需要的依赖。请运行：pip install hqdata[ricequant]开启对ricequant的支持。" | ||
| """rqdatac is not installed. | ||
|
|
||
| hqdata does not install dependencies you may not need by default. | ||
| Please run: pip install hqdata[ricequant] to enable ricequant support. | ||
| """ | ||
| ) from None |
Comment on lines
+207
to
+212
| click.echo(f"[{source}][stock-daily] ERROR: {e}", err=True) | ||
| click.echo( | ||
| f"[{source}][stock-daily] {written} day(s) already written. " | ||
| "Re-run the same command to resume.", | ||
| err=True, | ||
| ) |
Comment on lines
342
to
344
| assert_success(result) # the CLI reports the error but does not crash | ||
| assert "ERROR" in result.output | ||
| out_dir = tmp_path / "tushare" / "stock_daily" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.