Reports 1-1 of 1 Clear search Modify search
AdV-DAQ (Data collection)
swinkels - 18:55 Tuesday 19 July 2016 (34317) Print this report
Data flux

Now that most sub-systems are online, we should slowly think about the data flux in the DAQ system (to not cause DAQ instabilities like this afternoon), and the size of the frames written to disk (to make sure we have a large enough circular buffer on site). Attached script spits out some statistics about the frame files, to get an idea of the biggest producers of data. For now, a lot of intermediate channels are generated, which are nice for debugging, but which should probably be reduced to a minimum in the long term. It also shows some channels with prefixes that should probably be changed. Result today:

Statistics for frame file /virgoData/ffl/raw.ffl at gps = 1152982190 (2016-07-19 16:49:33 UTC)
       Sc: 16720000 byte/s,  418 channels (Sc_BS_F7_LVDT_H1, Sc_BS_F7_LVDT_H2, ...)
     SDB1:  7600655 byte/s,  187 channels (SDB1_LC_COIL_BL_H, SDB1_LC_COIL_BL_V, ...)
     SNEB:  7301759 byte/s,  371 channels (SNEB_B7_PD1_sample, SNEB_B7_PD2_sample, ...)
     SIB2:  6178088 byte/s,  358 channels (SIB2_B2_PD1_sample, SIB2_B2_PD2_sample, ...)
       TD:  5200000 byte/s,   66 channels (TD_SNEB_LC_LVDT_BL_H_err, TD_SNEB_LC_LVDT_BL_H_in_mag, ...)
     SWEB:  4800160 byte/s,  183 channels (SWEB_B8_PD1_sample, SWEB_B8_PD2_sample, ...)
      INJ:  4646806 byte/s,  159 channels (INJ_SFP_SERVO_SET, INJ_SFP_SERVO_ENBL, ...)
     SDB2:  4538481 byte/s,  358 channels (SDB2_B1p_PD1_sample, SDB2_B1p_PD2_sample, ...)
      ENV:  3833152 byte/s,  361 channels (ENV_METEO_RAIN, ENV_METEO_HU, ...)
      LSC:  2980000 byte/s,   75 channels (LSC_B1p_DC, LSC_B1s1_DC, ...)
      SAS:  2960000 byte/s,   74 channels (SAS_EIB_DAC_act0, SAS_EIB_DAC_act1, ...)
     SSFS:  1560004 byte/s,   22 channels (SSFS_B4_6MHz_I, SSFS_B4_6MHz_Q, ...)
      PSL:  1416032 byte/s,   30 channels (PSL_SL_XTAL2_TE_SET, PSL_SL_XTAL2_TE, ...)
      DAQ:  1364964 byte/s,   72 channels (DAQ_PYFLAGS_MEM, DAQ_PYFLAGS_ETIME, ...)
     SPRB:  1288909 byte/s,  168 channels (SPRB_B4_PD1_sample, SPRB_B4_PD2_sample, ...)
       Bs:  1124000 byte/s,   29 channels (Bs_IB_z, Bs_IMC_TRA, ...)
       Tt:  1040000 byte/s,   26 channels (Tt_SIB2B2_QD1_GALVO_ENBL, Tt_SIB2B2_QD1_GALVO_ON, ...)
       Sb:   840000 byte/s,   21 channels (Sb_GA_c1, Sb_GA_c2, ...)
       Sa:   814116 byte/s,  417 channels (Sa_BC_Gain_g_y, Sa_BC_Gain_yOn, ...)
      EIB:   660000 byte/s,   17 channels (EIB_Pr_SAS_CfgChange, EIB_Pr_SAS_Time, ...)
    GALVO:   640000 byte/s,   16 channels (GALVO_B2_QD1_DCh_corr, GALVO_B2_QD1_DCv_corr, ...)
    PSTAB:   560000 byte/s,   10 channels (PSTAB_AMP_CORR, PSTAB_PD1_AC, ...)
      SBE:   420000 byte/s,  210 channels (SBE_SDB2_ACT_F0H0raw_500Hz, SBE_SDB2_ACT_F0H1raw_500Hz, ...)
       Ti:   400000 byte/s,    1 channels (Ti_Gps, ...)
      TCS:   320072 byte/s,   26 channels (TCS_PR_RH2_CURR, TCS_PR_RH2_VOLT, ...)
       CA:   280000 byte/s,    6 channels (CA_BC_NOISE, CA_IB_SA_NOISE, ...)
      SAT:   260732 byte/s,  190 channels (SAT_WI_F0_Z_SET, SAT_WI_F0_X_SET, ...)
     ISYS:   260600 byte/s,    9 channels (ISYS_EER_dac_CfgChange, ISYS_EER_dac_Time, ...)
      BsX:   240000 byte/s,    6 channels (BsX_QF_DC, BsX_QF_h, ...)
       LP:   128000 byte/s,    4 channels (LP_SBE_SDB2_tmp1, LP_SNEB_B7_6MHz_I, ...)
      NEB:   120000 byte/s,    3 channels (NEB_Monitoring_CfgChange, NEB_Monitoring_Time, ...)
  CaNoise:   120000 byte/s,    3 channels (CaNoise_CfgChange, CaNoise_Time, ...)
     LNFS:   120000 byte/s,    3 channels (LNFS_Demod_CfgChange, LNFS_Demod_Time, ...)
      SDB:    80019 byte/s,   26 channels (SDB_EDB_CAM0_shutter_close_count, SDB_EDB_CAM0_shutter_open_count, ...)
      ISC:    80000 byte/s,    2 channels (ISC_Tpro_elapsed_time, ISC_Tpro_processed_packets, ...)
      PD3:    80000 byte/s,    1 channels (PD3_Pol, ...)
      PD1:    80000 byte/s,    1 channels (PD1_DC, ...)
      PD2:    80000 byte/s,    1 channels (PD2_RF, ...)
    Pstab:    80000 byte/s,    2 channels (Pstab_Pressure, Pstab_TE, ...)
  DAC1955:    40000 byte/s,    1 channels (DAC1955_Calib, ...)
DSPtoSSFS:    40000 byte/s,    1 channels (DSPtoSSFS_ssfs_dac_ch_INT, ...)
      VAC:    23451 byte/s, 8934 channels (VAC_TO_WI_MonitorCodeAuthorizationSpv, VAC_TO_WI_MonitorFullControlStSpv, ...)
      Daq:    17480 byte/s, 4370 channels (Daq_WEB_ADC_Moni0_Tolm_Send_RxTxErrorFlag, Daq_WEB_ADC_Moni0_Tolm_Send_Total_Max_Index, ...)
       Em:     4000 byte/s,    1 channels (Em_test, ...)
      PAY:     2572 byte/s,   59 channels (PAY_NE_CamCameraLatency, PAY_NE_CamImageTransferTime, ...)
       Qc:     1116 byte/s,  279 channels (Qc_Vacuum_MG..CB_BS_PR1_absent, Qc_Vacuum_MG..MC_PR1_absent, ...)
     HVAC:      952 byte/s,  238 channels (HVAC_loop_DET_ENBL, HVAC_loop_DET_error_max, ...)
       SA:      852 byte/s,  213 channels (SA_WI_M2_BL, SA_WI_M2_BR, ...)
       MC:      770 byte/s,   22 channels (MC_MIR_FRONTCameraLatency, MC_MIR_FRONTImageTransferTime, ...)
     IMMS:      452 byte/s,  113 channels (IMMS_MC_UPS2_ALRM_TIME, IMMS_MC_UPS2_ALRM, ...)
      INF:      408 byte/s,  102 channels (INF_LLAB_NEW_PRES, INF_MC_TUNNEL_LUX, ...)
      EDB:      362 byte/s,   10 channels (EDB_B1_CamCameraLatency, EDB_B1_CamImageTransferTime, ...)
       DQ:       20 byte/s,    5 channels (DQ_DUMMY_FLAG, DQ_WORK_HOURS, ...)
       MB:        4 byte/s,    1 channels (MB_ModbusVac_State, ...)
Total:    81348991 byte/s, 18281 channels
 

Non-image files attached to this report
Comments to this report:
swinkels - 10:31 Friday 22 July 2016 (34340) Print this report
Note that the above numbers are based on the nBits and sampleRate fields in the FrAdcData structure found in the frames. The size of the corresponding 300-second gwf file (V-raw-1152981900-300.gwf) is 11643795108 bytes, which corresponds to a rate of 38812650 byte/s written to disk. This would suggest that the frames are compressed to a little bit less than 50% of the nominal size. This is still almost a factor 2 larger than the foreseen data flux (21MB/s according to Michele).
swinkels - 18:50 Tuesday 28 February 2017 (36704) Print this report


We still need to do some effort to reduce the data flux a bit. I reran my old script to understand which subsystems are contributing the most:



Statistics for frame file /virgoData/ffl/raw.ffl at gps = 1172335390 (2017-02-28 16:42:52 UTC)

       Sc: 19600000 byte/s,  490 channels (Sc_BS_F7_LVDT_H1, Sc_BS_F7_LVDT_H2, ...)

     SDB2:  8141977 byte/s,  725 channels (SDB2_Photodiodes_runTime, SDB2_Photodiodes_setTime, ...)

     SWEB:  7492267 byte/s,  394 channels (SWEB_Photodiodes_runTime, SWEB_Photodiodes_setTime, ...)

      ENV:  6309466 byte/s,  486 channels (ENV_WE_DUST_5UM, ENV_WE_DUST_2UM, ...)

      LSC:  6080920 byte/s,  382 channels (LSC_BS_CORR_LSC_NOISE, LSC_BS_CORR_DARM_CORR, ...)

     SIB2:  5340832 byte/s,  517 channels (SIB2_RFC_PD2_DC_adcBias, SIB2_RFC_PD2_DC_adcSlope, ...)

      INJ:  4843472 byte/s,  260 channels (INJ_RSWP_Pos, INJ_RSWP_ReqPos, ...)

     SSFS:  3960048 byte/s,   36 channels (SSFS_B4_6MHz_I_NORM_10kHz, SSFS_B4_6MHz_phi0_offset, ...)

     SPRB:  3723029 byte/s,  370 channels (SPRB_Photodiodes_restart, SPRB_DBOX_SBE_mezz3_temp1, ...)

     SDB1:  3138341 byte/s,  199 channels (SDB1_DBOX_LC_mezz0_temp1, SDB1_DBOX_LC_mezz0_temp0, ...)

     SNEB:  3126222 byte/s,  384 channels (SNEB_Photodiodes_runTime, SNEB_Photodiodes_setTime, ...)

      PSL:  1416036 byte/s,   32 channels (PSL_SL_INJSL_TempControllerEER_State, PSL_SL_XTAL2_TE_SET, ...)

       Sa:   826000 byte/s,  413 channels (Sa_SR_BL_K_V_500Hz, Sa_SR_BR_LVDT1_500Hz, ...)

      TCS:   800204 byte/s,   72 channels (TCS_CO2_WI_OUT_Pos, TCS_CO2_WI_OUT_ReqPos, ...)

      BsX:   760000 byte/s,   19 channels (BsX_PZT_DH, BsX_PZT_DV, ...)

      DAQ:   726114 byte/s,   62 channels (DAQ_PYSFP_MEM, DAQ_PYSFP_ETIME, ...)

      ASC:   584016 byte/s,   73 channels (ASC_unit, ASC_Acl_runTime, ...)

    PSTAB:   560000 byte/s,   10 channels (PSTAB_AMP_CORR, PSTAB_PD1_AC, ...)

      CAL:   480076 byte/s,   31 channels (CAL_unit, CAL_WE_MIR_Z_CORR_WE_colornoise, ...)

      EDB:   440353 byte/s,   84 channels (EDB_DBOX_DetEEroom_mezz2_temp1, EDB_DBOX_DetEEroom_mezz2_temp0, ...)

     ISYS:   400800 byte/s,   12 channels (ISYS_Acl_CfgChange, ISYS_Acl_Time, ...)

      SBE:   325600 byte/s,  202 channels (SBE_SPRB_ACT_F0H0raw_500Hz, SBE_SPRB_ACT_F0H1raw_500Hz, ...)

      EIB:   240000 byte/s,    6 channels (EIB_PC_CfgChange, EIB_PC_Time, ...)

     EPRB:   200128 byte/s,   38 channels (EPRB_B4_PC_6MHz_LSB_PWR, EPRB_B4_PC_8MHz_LSB_PWR, ...)

     SDD1:   160000 byte/s,    4 channels (SDD1_LC_TX_fb, SDD1_LC_TY_fb, ...)

 CALnoise:   120012 byte/s,    6 channels (CALnoise_runTime, CALnoise_setTime, ...)

      NEB:   120008 byte/s,   13 channels (NEB_DBOX_TCS_PCAL_mezz0_temp1, NEB_DBOX_TCS_PCAL_mezz0_temp0, ...)

ISYSnoise:   120000 byte/s,    3 channels (ISYSnoise_CfgChange, ISYSnoise_Time, ...)

     LNFS:   120000 byte/s,    3 channels (LNFS_Demod_CfgChange, LNFS_Demod_Time, ...)

      SDB:    80016 byte/s,   22 channels (SDB_DBOX_LC_mezz2_temp1, SDB_DBOX_LC_mezz2_temp0, ...)

      ISC:    80000 byte/s,    2 channels (ISC_Tpro_elapsed_time, ISC_Tpro_processed_packets, ...)

 ENVnoise:    80000 byte/s,    3 channels (ENVnoise_CfgChange, ENVnoise_Time, ...)

       CA:    80000 byte/s,    1 channels (CA_NE_MIR_UL_MONI, ...)

    Pstab:    44000 byte/s,    2 channels (Pstab_Pressure, Pstab_TE, ...)

  DAC1955:    40000 byte/s,    1 channels (DAC1955_Calib, ...)

      VAC:    26730 byte/s, 9765 channels (VAC_VALVE_WI_RackStatus, VAC_VALVE_WI_RESETARD, ...)

      Daq:    23570 byte/s, 5988 channels (Daq_SWEB_Photodiodes_DAQ_1Hz_Tolm_Send_RxTxErrorFlag, Daq_SWEB_Photodiodes_DAQ_1Hz_Tolm_Send_Total_Max_Index, ...)

      PAY:    10584 byte/s,  120 channels (PAY_MC_Cam_CameraLatency, PAY_MC_Cam_FitAmpX, ...)

       DQ:     4572 byte/s, 1143 channels (DQ_META_LOCK_LOSS_INJ_MAIN, DQ_META_STATUS_INJ_MAIN, ...)

       Bs:     4000 byte/s,    1 channels (Bs_SL_D2R_DC_bug, ...)

      SQZ:     2000 byte/s,    1 channels (SQZ_PAY_SHAD_METER_500Hz, ...)

      SAT:     1468 byte/s,  367 channels (SAT_WI_satserverwi1_State, SAT_WI_F7_DC_ON, ...)

     HVAC:      960 byte/s,  240 channels (HVAC_loop_DET_ENBL, HVAC_loop_DET_error_max, ...)

       SA:      936 byte/s,  234 channels (SA_SR_MV.F3, SA_SR_MV.F2, ...)

       Qc:      713 byte/s,  180 channels (Qc_Vacuum_Qc_Vacuum_MC_Turbo, Qc_Vacuum_Qc_Vacuum_MC_BackPressure, ...)

     IMMS:      588 byte/s,  147 channels (IMMS_W1500_UPS2_ALRM_TIME, IMMS_W1500_UPS2_ALRM, ...)

      INF:      468 byte/s,  117 channels (INF_CEB_HALL_W_PRES, INF_TUNNEL_W_TE, ...)

      CEB:      138 byte/s,   96 channels (CEB_DBOX_PLATFORM_mezz0_temp1, CEB_DBOX_PLATFORM_mezz0_temp0, ...)

     META:       60 byte/s,   15 channels (META_WARM_LOCK_index, META_SUS_WI_index, ...)

      EER:       43 byte/s,   54 channels (EER_DBOX4_mezz2_temp1, EER_DBOX4_mezz2_temp0, ...)

     TCSR:       16 byte/s,    4 channels (TCSR_TE_WI_Chiller_Set, TCSR_TE_WI_Chiller, ...)

      WEB:        7 byte/s,   10 channels (WEB_DBOX_TCS_PCAL_mezz0_temp1, WEB_DBOX_TCS_PCAL_mezz0_temp0, ...)

       MB:        4 byte/s,    1 channels (MB_ModbusVac_State, ...)

Total:     80636801 byte/s, 23840 channels



Note that these numbers are for the un-compressed flux. The current frame size is around 7 GB for 200 seconds, so 35e6 byte/s. This suggest that frames can be compressed to a bit more than 40% of their original size.



This lists also hints at some channels that could be renamed to make things a bit more uniform and to avoid too many top-level prefixes, for example:




  • BsX_* -> INJ_BPC_*

  • PSTAB_* -> INJ_PSTAB_*,

  • SomeProcess_ElapsedTime -> DAQ_SomeProcess_ElapsedTime

  • SomeDaqBox_Temperature -> DAQ_SomeDaqBox_Temperature

  • Daq -> DAQ

  • [Sc, Sa, PAY, SAT, SA] -> ???

  • CA_* -> CAL_*

  • MB_* -> VAC_*,

  • IMMS_* -> INF_*?

  • SDD1 -> SDB1??

  • Bs_SL_D2R_DC_bug -> remove?

  • Qc -> DQ?

mours - 16:12 Wednesday 01 March 2017 (36716) Print this report

A more realistic check of the data rate and channel size and could be obtained with the FrDump command. This could provide the ranking of the compressed channel (the uncompressed information is sometime very wrong).

For instance:

FrDump -i /virgoData/ffl/raw.ffl -f 1172415200 -l 1 -d 3 -top 50 -c -1 -t "V1:LSC*"

Will provide the ranked list for the 50 bigger channels starting with V1:LSC at GPS=1172415200, as well as the total size (over 10 seconds since the frames are 10 seconds long).

masserot - 17:45 Wednesday 01 March 2017 (36720) Print this report

Here for the same GPS time, the sort according the compressed data rate instead of the uncompressed data rate.

    Bytes/s
  nb channels compressed uncompressed
V1:* 24023 34230478 119286544
V1:Sc_* 490 7489451 19605580
V1:SDB2_* 725 5336729 18935724
V1:SWEB_* 432 3927611 12937562
V1:SIB2_* 517 2981630 10870672
V1:ENV_* 486 2274262 6380458
V1:LSC_* 382 1877538 6085417
V1:SNEB_* 384 1727868 8570465
V1:SPRB_* 370 1585651 6358228
V1:INJ_* 260 1577322 5903227
V1:SDB1_* 199 972134 3140806
V1:EDB_* 84 760044 4222808
V1:PAY_* 139 692689 2643871
V1:Sa_* 413 468737 830924
V1:PSL_* 32 310177 1416395
V1:EPRB_* 38 308970 1120578
V1:BsX_* 19 288563 760204
V1:DAQ_* 62 245530 792375
V1:SBE_* 202 191506 328019
V1:SSFS_* 36 179535 3960482
V1:ASC_* 73 146084 584829
V1:TCS_* 72 131473 801039
V1:VAC_* 9765 131413 148755
V1:Daq_* 6114 101249 110752
V1:PSTAB_* 10 100765 560109
V1:ISYS_* 12 66800 400941
V1:CA_* 1 51602 80011
V1:EIB_* 6 51380 240069
V1:DQ_* 1143 19993 21861
V1:NEB_* 13 18558 120174
V1:SDB_* 22 13638 80289
V1:ISC_* 2 13534 80025
V1:SAT* 367 4783 5769
V1:HVAC_* 240 3228 3840
V1:SA_* 234 2914 3604
V1:Qc_* 180 2654 3268
V1:CAL_* 31 2513 480460
V1:IMMS_* 147 1943 2374
V1:INF_* 117 1780 1803
V1:CEB_* 96 1304 1341
V1:EER_* 54 699 699
V1:Bs_* 1 606 4011
V1:WEB_* 10 137 137

 

and the 30 first channels according their compressed data rate

    0;  4140490; 1.1984; 1.1984; V1:SIB2_B2_Cam_Zoom__Image; 1310720.00;  1;  3;
    1;  3614624; 1.0462; 2.2445; V1:SDB2_B5_Cam_Image; 1310720.00;  1;  3;
    2;  3382789; 0.9791; 3.2236; V1:SIB2_B2_Cam__Image; 1310720.00;  1;  3;
    3;  3039362; 0.8797; 4.1032; V1:SDB2_B1_Cam_Image; 1310720.00;  1;  3;
    4;  2720335; 0.7873; 4.8906; V1:SNEB_B7_Cam2_Image; 1310720.00;  1;  3;
    5;  2630902; 0.7614; 5.6520; V1:SDB2_B1p_Cam_Image; 1310720.00;  1;  3;
    6;  2431839; 0.7038; 6.3558; V1:SDB2_B1s2_Cam_Image; 1310720.00;  1;  3;
    7;  2226632; 0.6444; 7.0003; V1:EDB_B1_Cam_Image; 1310720.00;  1;  3;
    8;  1841245; 0.5329; 7.5332; V1:SWEB_LC_LVDT_FR_H_in_raw; 50000.00;  3;  1;
    9;  1841155; 0.5329; 8.0661; V1:SWEB_LC_LVDT_FL_H_in_raw; 50000.00;  3;  1;
   10;  1840848; 0.5328; 8.5988; V1:SWEB_LC_LVDT_FL_V_in_raw; 50000.00;  3;  1;
   11;  1840667; 0.5327; 9.1316; V1:SWEB_LC_LVDT_FR_V_in_raw; 50000.00;  3;  1;
   12;  1840657; 0.5327; 9.6643; V1:SWEB_LC_LVDT_BL_H_in_raw; 50000.00;  3;  1;
   13;  1840035; 0.5326; 10.1969; V1:SWEB_LC_LVDT_BR_V_in_raw; 50000.00;  3;  1;
   14;  1839384; 0.5324; 10.7292; V1:SWEB_LC_LVDT_BR_H_in_raw; 50000.00;  3;  1;
   15;  1837827; 0.5319; 11.2611; V1:SWEB_LC_LVDT_BL_V_in_raw; 50000.00;  3;  1;
   16;  1832429; 0.5303; 11.7915; V1:SDB2_LC_LVDT_BL_H_FR_V_in_raw; 50000.00;  3;  1;
   17;  1743513; 0.5046; 12.2961; V1:PAY_MC_Cam_Image; 327680.00;  1;  5;
   18;  1697269; 0.4912; 12.7873; V1:PAY_NI_Cam_Image; 327680.00;  1;  5;
   19;  1696201; 0.4909; 13.2782; V1:PAY_WI_Cam_Image; 327680.00;  1;  5;
   20;  1682231; 0.4869; 13.7651; V1:PAY_NE_Cam_Image; 327680.00;  1;  5;
   21;  1653914; 0.4787; 14.2438; V1:SWEB_B8_Cam2_Image; 1310720.00;  1;  3;
   22;  1478833; 0.4280; 14.6718; V1:SNEB_B7_DC_40kHz; 40000.00;  3;  1;
   23;  1448107; 0.4191; 15.0909; V1:SWEB_B8_DC_40kHz; 40000.00;  3;  1;
   24;  1437776; 0.4161; 15.5071; V1:SWEB_B8_Cam1_Image; 1310720.00;  1;  3;
   25;  1105701; 0.3200; 15.8271; V1:SPRB_B4_Cam1_Image; 1310720.00;  1;  3;
   26;   743382; 0.2152; 16.0422; V1:ENV_NOISE_CEB_DAQroom; 20000.00;  3;  1;
   27;   743168; 0.2151; 16.2573; V1:SDB2_B1s1_PD1_DC; 20000.00;  3;  1;
   28;   742371; 0.2149; 16.4722; V1:SIB2_RFC_PD2_DC; 20000.00;  3;  1;
   29;   740678; 0.2144; 16.6866; V1:SDB2_B1s2_PD1_DC; 20000.00;  3;  1;
   30;   740416; 0.2143; 16.9008; V1:SDB2_B1s1P_PD1_DC; 20000.00;  3;  1;
 

Search Help
×

Warning

×