Virtual Solar Observatory
Published April 19, 2015 | Version v1
Conference paper Open

Distributing Solar Data: Minimizing Wasted Bandwidth

Authors/Creators

  • 1. NASA/GSFC

Description

Abstract—The solar physics community archives most of its data using the FITS file format. We discuss advantages and disadvantages of FITS files, particularly as related to transfer of data across the internet. We suggest small changes to archives that manage FITS files, and techniques that could be used by sites that mirror these files. We discuss the need for a more holistic approach in designing and implementing solar data systems.

Notes

This is a paper that I had originally submitted for the 2014 SABiD (Solar Astronomy: Big Data) workshop. After getting frustrated with their paper submision process, I withdrew the paper. There are some strange formatting artifacts, as the template that IEEE provided had a strange interaction with OpenOffice, and the 'V' in headings have a strange font.

Files

jhourcle_2014SABID_data_transport.pdf

Files (187.9 kB)

Name Size Download all
md5:30d3c68d6437501a78317fc6a23a88a6
187.9 kB Preview Download

Additional details

References

  • [1]FCC (2014 Aug. 1) “Proceeding 14-126: Inquiry Concerning the Deployment of Advanced Telecommunications Capability to All Americans in a Reasonable and Timely Fashion” [Online] Available: http://apps.fcc.gov/ecfs/proceeding/view?name=14-126
  • [2]A. Beard, B. Cowan, and A. Ferayorni, “DKIST visible broadband imager data processing pipeline”, Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91521J (July 18, 2014); DOI: http://dx.doi.org/10.1117/12.2057122
  • [3]W. Pence, L. Chiappetti, C. Page, R. Shaw and E. Stobie, “[2] Definition of the Flexible Image Transport System (FITS), version 3.0”, Astronomy & Astrophysics Vol. 524 Abstract #A42 (2010). DOI: http://dx.doi.org/10.1051/0004-6361/201015362
  • [4]F. Ochsenbein and M. Taylor, Eds, (2013, Sept. 20) “VOTable Format Definition, Version 1.3” [Online] Available: http://www.ivoa.net/documents/VOTable/
  • [5]R. Bogart, J. Aloise, A. Amezcua, J. Hourclé, P. Scherrer, J. Spencer, F. Suárez Solá, “NetDRMS: A Shared Data Management System for SDO and VSO” in American Astronomical Society, SPD meeting #40, abstract #17.05; Bulletin of the American Astronomical Society, Vol. 41, p.842. http://adsabs.harvard.edu/abs/2009SPD....40.1705B
  • [6]HEASARC, NASA (n.d) “FITS File Verifier” [Online] Avalable: http://fits.gsfc.nasa.gov/fits_verify.html
  • [7]J. Hsu, P. Barrett, C. Hanley, J. Taylor, M. Droettboom, and E. Bray (n.d.) “Executable Scripts — PyFITS 3.3 Documentation” [Online] Available: https://pythonhosted.org/pyfits/users_guide/users_scripts.html
  • [8]J. Heirbaut (2011, Sept.) “JojoDiff - diff utility for binary files” [Online] Available : http://jojodiff.sourceforge.net/
  • [9]R. Seaman, “FITS Checksum Verification in the NOAO Archive” in Astronomical Data Analysis Software and Systems IV, ASP Conference Series, Vol. 77, 1995. Avalable: http://adass.org/adass/proceedings/adass94/seamanr.html ;
  • [10]R.L. Seaman, W.D. Pence, and A. H. Rots (2002, May 23) “FITS Checksum Protocol” [Online] Available: http://fits.gsfc.nasa.gov/registry/checksum.html
  • [11]R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee (1999 June), “Hypertext Transfer Protocol – HTTP/1.1” [Online] Available: http://www.w3.org/Protocols/rfc2616
  • [12]F. Hill, R. Bogart, A. Davey, G. Dimitoglou, J. Gurman, J. Hourclé, P. Martens, F. Suárez Solá, K. Tian, S. Wampler, and K. Yoshimura (2004, Sept.). “The Virtual Solar Observatory: status and initial operational experience”. In Astronomical Telescopes and Instrumentation (pp. 163-169). International Society for Optics and Photonics. http://dx.doi.org/10.1117/12.552842
  • [13]J. Hourclé, (2014, June) “Data Citation in Astronomy”, in Library and Information Services in Astronomy VII, conference held 17-20 June 2014 in Naples, Italy. DOI: http://dx.doi.org/10.5281/zenodo.10505
  • [14]G. Taffoni, A. Barisani, C. Vuerli, W. Pence, F. Pasian, “Managing Astronomical Data in the Grid through CFITSIO”, in Astronomical Data Analysis Software and Systems XVI ASP Conference Series, Vol. 376, proceedings of the conference held 15-18 October 2006 in Tucson, Arizona, USA. Edited by Richard A. Shaw, Frank Hill and David J. Bell, p.89. http://adsabs.harvard.edu/abs/2007ASPC..376...89T
  • [15]W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, “The Globus Striped GridFTP Framework and Server” in Proceedings of the 2005 ACM/IEEE conference on Supercomputing, Seattle, WA, p. 54. Available: http://www.cresco.enea.it/SC05/schedule/pdf/pap299.pdf
  • [16]RENCI (2014, Aug. 20), “iRODS 4.0.3 – Manual” [Online] Available: ftp://ftp.renci.org/pub/irods/releases/4.0.3/irods-manual-4.0.3.pdf
  • [17]R. Rank, “Enterprise IT support for NOAA archives” in Geoscience and Remote Sensing Symposium, Barcelona, Spain, 2007, pp. 4025-4028. http://dx.doi.org/10.1109/IGARSS.2007.4423732
  • [18]C. Mattmann, D. Crichton, J. Hughes, P. Ramirez, C. Goodale, and A. Hart, “The Apache OODT Project: An Introduction”, abstract #IN11D-1484 in American Geophysical Union Fall Meeting, San Francisco, 2012. http://adsabs.harvard.edu/abs/2012AGUFMIN11D1484M
  • [19]J. Hughes, D. Crichton, S. Kelly, S. Lavoie and R. Walker, “A Data Grid Framework for Sharing Data Across Distributed Heterogeneous Repositories”, abstract #U22A-0003 in American Geophysical Union Fall Meeting, San Francisco, CA, 2003. http://adsabs.harvard.edu/abs/2003AGUFM.U22A0003H