반응형
0. 실행 환경
AWS EC2 t2.xlarge
OS : Red Hat 9.1
Python : 3.9
Airflow : 2.5.0
1. Code
with DAG(
is_api_available = HttpSensor(
task_id = 'is_api_available',
http_conn_id = 'spotify_api',
# method="GET",
headers = {
# 'Accept': 'application/json',
# 'Content-Type': 'application/json',
'Authorization':
'Bearer <MYTOKEN>',
},
request_params = {
'q': 'BTS',
'type': 'artist',
'limit': '1',
},
method="GET",
endpoint='v1/search'
)curl, request로 하면 결과가 나오는데 task 실행시키니까 400 에러가난다.
HttpSensor, SimpleHttpOperator 로 해봐도 안된다.
response_check=False를 넣어도 안된다.
providers/http/hooks/http.py, requests/models.py를봐도 response code를 확인하는거 밖에 없는데 원인을 모르겠다.
task정의에서 header를 제거해도 같은 오류가난다.
2. Connection
3. API Test
1) Curl 결과
2) Requests
import requests
headers = {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': 'Bearer <MYTOKEN>',
}
params = {
'q': 'BTS',
'type': 'artist',
'limit': '1',
}
response = requests.get('https://api.spotify.com/v1/search', params=params, headers=headers)
print(response)
4. Error
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1
[2023-01-01 08:52:44,836] {http.py:122} INFO - Poking: v1/search
[2023-01-01 08:52:44,839] {base.py:73} INFO - Using connection ID 'spotify_api' for task execution.
[2023-01-01 08:52:44,840] {http.py:150} INFO - Sending 'GET' to url: https://api.spotify.com/v1/search
[2023-01-01 08:52:44,988] {http.py:163} ERROR - HTTP error: Bad Request
[2023-01-01 08:52:44,989] {http.py:164} ERROR - {
"error": {
"status": 400,
"message": "Only valid bearer authentication supported"
}
}
[2023-01-01 08:52:44,989] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 161, in check_response
response.raise_for_status()
File "/opt/anaconda/anaconda3/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/sensors/base.py", line 199, in execute
poke_return = self.poke(context)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 137, in poke
raise exc
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 124, in poke
response = hook.run(
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 151, in run
return self.run_and_check(session, prepped_request, extra_options)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 204, in run_and_check
self.check_response(response)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 165, in check_response
raise AirflowException(str(response.status_code) + ":" + response.reason)
airflow.exceptions.AirflowException: 400:Bad Request
[2023-01-01 08:52:44,990] {taskinstance.py:1322} INFO - Marking task as FAILED. dag_id=nft-pipeline, task_id=is_api_available, execution_date=20230101T085244, start_date=, end_date=20230101T085244
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 161, in check_response
response.raise_for_status()
File "/opt/anaconda/anaconda3/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/__main__.py", line 39, in main
args.func(args)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 52, in command
return func(*args, **kwargs)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 108, in wrapper
return f(*args, **kwargs)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 576, in task_test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1673, in run
self._run_raw_task(
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
return func(*args, **kwargs)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1378, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1524, in _execute_task_with_callbacks
result = self._execute_task(context, task_orig)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1585, in _execute_task
result = execute_callable(context=context)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/sensors/base.py", line 199, in execute
poke_return = self.poke(context)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 137, in poke
raise exc
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 124, in poke
response = hook.run(
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 151, in run
return self.run_and_check(session, prepped_request, extra_options)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 204, in run_and_check
self.check_response(response)
File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 165, in check_response
raise AirflowException(str(response.status_code) + ":" + response.reason)
airflow.exceptions.AirflowException: 400:Bad Request참고 자료
Airflow : providers/http/hooks/http.py
Python : requests/models.py
Airflow : HttpSensor
CurlConverter
728x90
반응형
