Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
A
Amazon-Selection-Data
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
abel_cjy
Amazon-Selection-Data
Commits
1452459f
Commit
1452459f
authored
Jun 01, 2026
by
fangxingjun
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
no message
parent
8e098cba
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
18 additions
and
11 deletions
+18
-11
wf_month_control.py
Pyspark_job/listen_program/wf_month_control.py
+18
-11
No files found.
Pyspark_job/listen_program/wf_month_control.py
View file @
1452459f
...
@@ -105,14 +105,21 @@ if __name__ == '__main__':
...
@@ -105,14 +105,21 @@ if __name__ == '__main__':
# for site_name in ['us', 'uk', 'de']:
# for site_name in ['us', 'uk', 'de']:
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_asin', wf_type="spider")
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_asin', wf_type="spider")
site_name
=
'us'
site_name
=
sys
.
argv
[
1
]
# 参数1:站点
# 同步st搜索词
date_type
=
sys
.
argv
[
2
]
# 参数2:类型:day/week/4_week/month/quarter
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_st'
,
wf_type
=
"spider"
)
date_info
=
sys
.
argv
[
3
]
# 参数3:年-月-日/年-周/年-月/年-季, 比如: 2022-1
# 抓完搜索词+同步asin -- st抓取完计算数量+1 + 重置asin抓取
spider_name
=
sys
.
argv
[
4
]
# 参数4:spider_name名称对应的值, 其实也是爬虫任务流
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_asin'
,
wf_type
=
"spider"
)
wf_type
=
sys
.
argv
[
5
]
# 参数4:spider或cal, 判断执行的类型
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_st'
,
wf_type
=
"cal"
)
wf_month_control
(
site_name
=
site_name
,
date_type
=
date_type
,
date_info
=
date_info
,
spider_name
=
spider_name
,
wf_type
=
wf_type
)
# 同步fd -- 重置asin抓取+st抓取完数量+1
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_fd'
,
wf_type
=
"spider"
)
# site_name = 'us'
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_fd'
,
wf_type
=
"cal"
)
# # 同步st搜索词
# 抓完asin+计算全流程 -- 更改asin计算全流程数量+1
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_st', wf_type="spider")
wf_month_control
(
site_name
=
site_name
,
date_type
=
'month'
,
date_info
=
'2026-06'
,
spider_name
=
f
'{site_name}_spider_asin'
,
wf_type
=
"cal"
)
# # 抓完搜索词+同步asin -- st抓取完计算数量+1 + 重置asin抓取
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_asin', wf_type="spider")
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_st', wf_type="cal")
# # 同步fd -- 重置asin抓取+st抓取完数量+1
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_fd', wf_type="spider")
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_fd', wf_type="cal")
# # 抓完asin+计算全流程 -- 更改asin计算全流程数量+1
# wf_month_control(site_name=site_name, date_type='month', date_info='2026-06', spider_name=f'{site_name}_spider_asin', wf_type="cal")
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment