[2026-05-13T02:02:17.559188] Job 157042 queued by schedule:2026-05-13T02:02:00 Connection: bauliitalia (instagram) Sync template: Instagram Graph - Webhook Reconcile Queue class: foreground Procrastinate queue: kpiper Command: python instagram_graph_pipeline.py [2026-05-13T02:02:17.665643] Queued via Procrastinate as job 155893 (queue=kpiper, priority=50). [2026-05-13T05:21:55.888541] Starting job 157042 for connection bauliitalia (instagram) / sync template Instagram Graph - Webhook Reconcile (type=dlt) Resolved command: cd /srv/kpiper && python instagram_graph_pipeline.py Injected profile context env vars (tokens exported: 1). Pipeline env overrides applied: COMMENTS_MEDIA_LOOKBACK_DAYS, DOWNLOAD_ASSETS, ENABLE_ACCOUNT_OBJECTS, ENABLE_ASSETS, ENABLE_COMMENTS, ENABLE_COMMENT_REPLIES, ENABLE_CONTENT_PUBLISHING_LIMITS, ENABLE_MEDIA_CHILD_OBJECTS, ENABLE_MEDIA_INSIGHTS, ENABLE_MEDIA_OBJECTS, ENABLE_STORY_INSIGHTS, ENABLE_STORY_OBJECTS, ENABLE_USER_INSIGHTS, ENABLE_WEBHOOK_SUBSCRIBED_APPS, IG_PAGE_ID, IG_USER_ID, MEDIA_INCREMENTAL_LOOKBACK_DAYS, MEDIA_PAGINATION, SOURCES__INSTAGRAM_GRAPH_REST_API_SOURCE__ACCESS_TOKEN, SYNC_MODE Refreshed Meta Page Token at 2026-05-13T05:21:55.861883. Forced unbuffered Python output for child process. /usr/local/lib/python3.11/site-packages/dlt/pipeline/progress.py:27: UserWarning: psutil dependency is not installed and memory stats will not be available. Add psutil to your environment or pass dump_system_stats=False to disable this warning. return log() ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/2 (0.0%) | Time: 0.00s | Rate: 0.00/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/2 (0.0%) | Time: 0.00s | Rate: 0.00/s media_objects: 1 | Time: 0.00s | Rate: 102300.10/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/2 (0.0%) | Time: 0.66s | Rate: 0.00/s media_objects: 1 | Time: 0.65s | Rate: 1.54/s media_comments: 1 | Time: 0.00s | Rate: 116508.44/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/2 (0.0%) | Time: 1.74s | Rate: 0.00/s media_objects: 2 | Time: 1.74s | Rate: 1.15/s media_comments: 1 | Time: 1.09s | Rate: 0.92/s media_comment_replies: 1 | Time: 0.00s | Rate: 97541.95/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/2 (0.0%) | Time: 3.39s | Rate: 0.00/s media_objects: 4 | Time: 3.39s | Rate: 1.18/s media_comments: 1 | Time: 2.73s | Rate: 0.37/s media_comment_replies: 3 | Time: 1.65s | Rate: 1.82/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 5.10s | Rate: 0.20/s media_objects: 4 | Time: 5.10s | Rate: 0.78/s media_comments: 1 | Time: 4.45s | Rate: 0.22/s media_comment_replies: 5 | Time: 3.36s | Rate: 1.49/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 7.56s | Rate: 0.13/s media_objects: 4 | Time: 7.55s | Rate: 0.53/s media_comments: 1 | Time: 6.90s | Rate: 0.14/s media_comment_replies: 7 | Time: 5.82s | Rate: 1.20/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 9.36s | Rate: 0.11/s media_objects: 4 | Time: 9.36s | Rate: 0.43/s media_comments: 1 | Time: 8.71s | Rate: 0.11/s media_comment_replies: 9 | Time: 7.62s | Rate: 1.18/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 10.90s | Rate: 0.09/s media_objects: 4 | Time: 10.90s | Rate: 0.37/s media_comments: 1 | Time: 10.25s | Rate: 0.10/s media_comment_replies: 10 | Time: 9.16s | Rate: 1.09/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 12.18s | Rate: 0.08/s media_objects: 4 | Time: 12.17s | Rate: 0.33/s media_comments: 1 | Time: 11.52s | Rate: 0.09/s media_comment_replies: 12 | Time: 10.43s | Rate: 1.15/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 13.63s | Rate: 0.07/s media_objects: 4 | Time: 13.63s | Rate: 0.29/s media_comments: 1 | Time: 12.98s | Rate: 0.08/s media_comment_replies: 16 | Time: 11.89s | Rate: 1.35/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 15.41s | Rate: 0.06/s media_objects: 4 | Time: 15.41s | Rate: 0.26/s media_comments: 1 | Time: 14.75s | Rate: 0.07/s media_comment_replies: 17 | Time: 13.67s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 17.30s | Rate: 0.06/s media_objects: 4 | Time: 17.30s | Rate: 0.23/s media_comments: 1 | Time: 16.65s | Rate: 0.06/s media_comment_replies: 21 | Time: 15.56s | Rate: 1.35/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 19.28s | Rate: 0.05/s media_objects: 4 | Time: 19.27s | Rate: 0.21/s media_comments: 1 | Time: 18.62s | Rate: 0.05/s media_comment_replies: 23 | Time: 17.53s | Rate: 1.31/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 21.34s | Rate: 0.05/s media_objects: 4 | Time: 21.33s | Rate: 0.19/s media_comments: 1 | Time: 20.68s | Rate: 0.05/s media_comment_replies: 25 | Time: 19.60s | Rate: 1.28/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 23.48s | Rate: 0.04/s media_objects: 4 | Time: 23.48s | Rate: 0.17/s media_comments: 1 | Time: 22.83s | Rate: 0.04/s media_comment_replies: 27 | Time: 21.74s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 25.21s | Rate: 0.04/s media_objects: 4 | Time: 25.20s | Rate: 0.16/s media_comments: 1 | Time: 24.55s | Rate: 0.04/s media_comment_replies: 29 | Time: 23.47s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 26.67s | Rate: 0.04/s media_objects: 4 | Time: 26.66s | Rate: 0.15/s media_comments: 1 | Time: 26.01s | Rate: 0.04/s media_comment_replies: 32 | Time: 24.93s | Rate: 1.28/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 27.79s | Rate: 0.04/s media_objects: 4 | Time: 27.78s | Rate: 0.14/s media_comments: 1 | Time: 27.13s | Rate: 0.04/s media_comment_replies: 33 | Time: 26.04s | Rate: 1.27/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 29.53s | Rate: 0.03/s media_objects: 4 | Time: 29.52s | Rate: 0.14/s media_comments: 1 | Time: 28.87s | Rate: 0.03/s media_comment_replies: 35 | Time: 27.78s | Rate: 1.26/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 31.63s | Rate: 0.03/s media_objects: 4 | Time: 31.63s | Rate: 0.13/s media_comments: 1 | Time: 30.98s | Rate: 0.03/s media_comment_replies: 37 | Time: 29.89s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 33.20s | Rate: 0.03/s media_objects: 4 | Time: 33.20s | Rate: 0.12/s media_comments: 1 | Time: 32.55s | Rate: 0.03/s media_comment_replies: 39 | Time: 31.46s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 35.11s | Rate: 0.03/s media_objects: 4 | Time: 35.11s | Rate: 0.11/s media_comments: 1 | Time: 34.46s | Rate: 0.03/s media_comment_replies: 42 | Time: 33.37s | Rate: 1.26/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 37.19s | Rate: 0.03/s media_objects: 4 | Time: 37.18s | Rate: 0.11/s media_comments: 1 | Time: 36.53s | Rate: 0.03/s media_comment_replies: 45 | Time: 35.45s | Rate: 1.27/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 42.57s | Rate: 0.02/s media_objects: 4 | Time: 42.56s | Rate: 0.09/s media_comments: 1 | Time: 41.91s | Rate: 0.02/s media_comment_replies: 46 | Time: 40.83s | Rate: 1.13/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 44.63s | Rate: 0.02/s media_objects: 4 | Time: 44.62s | Rate: 0.09/s media_comments: 1 | Time: 43.97s | Rate: 0.02/s media_comment_replies: 49 | Time: 42.89s | Rate: 1.14/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 45.71s | Rate: 0.02/s media_objects: 4 | Time: 45.70s | Rate: 0.09/s media_comments: 1 | Time: 45.05s | Rate: 0.02/s media_comment_replies: 51 | Time: 43.96s | Rate: 1.16/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 47.03s | Rate: 0.02/s media_objects: 4 | Time: 47.03s | Rate: 0.09/s media_comments: 1 | Time: 46.38s | Rate: 0.02/s media_comment_replies: 52 | Time: 45.29s | Rate: 1.15/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 48.63s | Rate: 0.02/s media_objects: 4 | Time: 48.62s | Rate: 0.08/s media_comments: 1 | Time: 47.97s | Rate: 0.02/s media_comment_replies: 58 | Time: 46.88s | Rate: 1.24/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 54.08s | Rate: 0.02/s media_objects: 4 | Time: 54.08s | Rate: 0.07/s media_comments: 1 | Time: 53.43s | Rate: 0.02/s media_comment_replies: 60 | Time: 52.34s | Rate: 1.15/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/2 (50.0%) | Time: 55.19s | Rate: 0.02/s media_objects: 4 | Time: 55.18s | Rate: 0.07/s media_comments: 1 | Time: 54.53s | Rate: 0.02/s media_comment_replies: 61 | Time: 53.44s | Rate: 1.14/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 2/2 (100.0%) | Time: 55.20s | Rate: 0.04/s media_objects: 4 | Time: 55.20s | Rate: 0.07/s media_comments: 1 | Time: 54.55s | Rate: 0.02/s media_comment_replies: 61 | Time: 53.46s | Rate: 1.14/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/1 (0.0%) | Time: 0.00s | Rate: 0.00/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 0/1 (0.0%) | Time: 0.01s | Rate: 0.00/s _dlt_pipeline_state: 1 | Time: 0.00s | Rate: 139810.13/s ------------------- Extract instagram_graph_rest_api_source -------------------- Resources: 1/1 (100.0%) | Time: 0.02s | Rate: 53.28/s _dlt_pipeline_state: 1 | Time: 0.01s | Rate: 73.20/s ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 0/4 (0.0%) | Time: 0.00s | Rate: 0.00/s ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 0/4 (0.0%) | Time: 0.03s | Rate: 0.00/s media_objects: 0 | Time: 0.00s | Rate: 0.00/s ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 1/4 (25.0%) | Time: 0.04s | Rate: 27.57/s media_objects: 4 | Time: 0.01s | Rate: 587.87/s media_comments: 0 | Time: 0.00s | Rate: 0.00/s ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 2/4 (50.0%) | Time: 0.04s | Rate: 48.68/s media_objects: 4 | Time: 0.01s | Rate: 344.28/s media_comments: 1 | Time: 0.00s | Rate: 207.45/s media_comment_replies: 0 | Time: 0.00s | Rate: 0.00/s ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 3/4 (75.0%) | Time: 0.06s | Rate: 51.92/s media_objects: 4 | Time: 0.03s | Rate: 141.29/s media_comments: 1 | Time: 0.02s | Rate: 46.49/s media_comment_replies: 61 | Time: 0.02s | Rate: 3652.95/s _dlt_pipeline_state: 0 | Time: 0.00s | Rate: 0.00/s 2026-05-13 05:22:57,941|[WARNING]|7683|139859167716096|dlt|validate.py|verify_normalized_table:113|In schema `instagram_graph_rest_api_source`: The following columns in table 'media_comments' did not receive any data during this load and therefore could not have their types inferred: - parent_comment_id Unless type hints are provided, these columns will not be materialized in the destination. One way to provide type hints is to use the 'columns' argument in the '@dlt.resource' decorator. For example: @dlt.resource(columns={'parent_comment_id': {'data_type': 'text'}}) ------- Normalize instagram_graph_rest_api_source in 1778649722.5353324 -------- Files: 4/4 (100.0%) | Time: 0.07s | Rate: 58.65/s media_objects: 4 | Time: 0.04s | Rate: 103.28/s media_comments: 1 | Time: 0.03s | Rate: 31.32/s media_comment_replies: 61 | Time: 0.03s | Rate: 2249.47/s _dlt_pipeline_state: 1 | Time: 0.01s | Rate: 95.91/s ---------- Load instagram_graph_rest_api_source in 1778649722.5353324 ---------- Jobs: 0/4 (0.0%) | Time: 0.00s | Rate: 0.00/s ---------- Load instagram_graph_rest_api_source in 1778649722.5353324 ---------- Jobs: 6/7 (85.7%) | Time: 1.04s | Rate: 5.78/s ---------- Load instagram_graph_rest_api_source in 1778649722.5353324 ---------- Jobs: 7/7 (100.0%) | Time: 1.09s | Rate: 6.42/s Pipeline instagram_graph_pipeline load step completed in 1.09 seconds 1 load package(s) were loaded to destination postgres and into dataset instagram_graph The postgres destination used postgresql://kpiper_dlt_app:***@[/cloudsql/pacelab-487114:europe-west4:pacelab-pg18-ew4]:5432/kpiper_data location to store data Load package 1778649722.5353324 is LOADED and contains no failed jobs [2026-05-13T05:22:59.770666] Command completed successfully. [2026-05-13T05:22:59.771004] Pipeline status payload: {"backfill_complete": true, "backfill_progress_mode": "incremental", "ig_user_id": "17841402185661173", "sync_mode": "incremental"}