[EN] [JP]
Last update: 19 May 2016, by A. Higuchi, CEReS Chiba University, Japan
Release note of "HIMAWARI 8" gridded data for full-disk (FD) observation mode

Atsushi HIGUCHI: Center for Environmental Remote Sensing (CEReS), Chiba University, Japan
Hideaki TAKENAKA: JAXA EORC / CEReS, Chiba University (visiting Asso. Prof.)
Koichi TOYOSHIMA: CEReS, Chiba University, Japan

Center for Environmental Remote Sensing (CEReS) released gridded-processed (lat-lon coordinated and geo-corrected processed) data of Himawari 8 geostationary meteorological satellite (H8) operated by Japan Meteorological Agency (JMA).

1. Gridded data specifications
1.1. Gridded data open-access server (ftp server)
Users can download as "anonymous user (anonymous ftp)" via an ftp server as below;

ftp://hmwr829gr.cr.chiba-u.ac.jp/

after anonymous login, please change directory as;
gridded/FD/V20151105/
Directly, ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20151105/
*V20151105 represents current program package version of whose processing from Himawari Standard data (HS) to our gridded data. It will be updated as newer (latest) version of program package in the future. However, currently we do not plan to update programs (as of 27 Apr 2016).

1.2. Gridded data stored sub-directories rule
The ftp server as shown in 1.1, automatically generates directories and sub-directories as below rules.
YYYYMM
YYYY: Year (4 digit) e.g., Year of 2015 -> 2015; MM: Month (always 2 digit) e.g., August -> 08. Each YYYYMM directory has four sub-directories as below.
 EXT VIS SIR TIR
Gridded data specifications stored in each sub-directory will be described in 1.3.

1.3. Gridded data specification and relation between CEReS released gridded data and JMA official Himawari 8/9 bands
Our CEReS released gridded data has different band naming rule with JMA official Himawari 8/9 bands names. Table 1 shows the relation between our gridded data and JMA's Himawari 8/9 bands.

Table 1. Relation between CEReS gridded data and JMA's Himawari 8/9 bands
CEReS gridded data JMA's H8, 9 band Pixel x Line gridded resolution
EXT 01 Band 03 (0.64 μ m) 24000 x 24000 0.005 degree (approx. 500 m)
VIS 01 Band 01 (0.47 μ m) 12000 x 12000 0.01 degree (approx. 1km)
02 Band 02 (0.51 μ m)
03 Band 04 (0.86 μ m)
SIR 01 Band 05 (1.6 μ m) 6000 x 6000 0.02 degree (approx. 2km)
02 Band 06 (2.3 μ m)
TIR 01 Band 13 (10.4 μ m) 6000 x 6000 0.02 degree (approx. 2km)
02 Band 14 (11.2 μ m)
03 Band 15 (12.4 μ m)
04 Band 16 (13.3 μ m)
05 Band 07 (3.9 μ m)
06 Band 08 (6.2 μ m)
07 Band 09 (6.9 μ m)
08 Band 10 (7.3 μ m)
09 Band 11 (8.6 μ m)
10 Band 12 (9.6 μ m)

Common Specifications: Covered area: 85 E -- 205 E (155 W), 60 N -- 60 S (same as our gridded data for MTSAT2)
2byte "unsigned short" binary data without any header and footer, and "big endian" data order
Data reading orders: West to East (left to right) and North to South (up to down).
Stored binary data is: HS data thenselves (CCT count like).

All bands data separately compressed by bzip2 (*.bz2).
[IMPORTANT] If user try to download ALL BANDS DATA, users storage would be required at least 3 TB! (even though compressed files!). We strongly recommended that only used bands data for analysis would be downloaded by using CUI commands, such as "wget", "ncftp", etc with shell script programming.

1.4. File name rules
File name specifications are as follows;
YYYYMMDDHHMN.XXX.ZZ.fld.geoss.bz2
YYYY: Year (4 digit) e.g., Year of 2015 -> 2015; MM: Month (always 2 digit) e.g., August -> 08.
DD: Day (always 2 digit) e.g., 12th -> 12; HH: Hour (always 2 digit; Unit is UTC) e.g., 02 UTC -> 02;
MN: Minute (always 2 digit; Unit is UTC) e.g., 10 minitues -> 10;
Time stamp in file name represents "observation start time (start time to scan)".
XXX: CEReS gridded data wavelengh-divided list. i.e. ext, vis, sir, and tir.
[Notice] Sub-directory names are all capital (EXT, VIS, SIR, and TIR). But in file name, these are expressed as small letters.
ZZ: CEReS gridded band number as listed in Table 1 (always 2 digit).
fld: FD version. geoss: no meaning. Simply CEReS geostationary gridded data's name rule.

1.5. Convert from raw digital number to physical variables (Tbb, spectral reflectance)
Converting from raw ditital data into physical variables such as brightness temperature (Tbb for thermal-IR, TIR), and spectral reflectances for visible, near-IR (EXT, VIS) and shortwave-IR (SIR) by the calibration tables of which produced by JMA, users have to need data processing via programing. We made sample programs (F90 and C) and calibration tables as a tarboll of count2tbb_v101.tgz. Same tarboll can download at; ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/support/
We check sample programs under the Linux (CentOS 6) and MacOS (OS X 10.11 C only) only. In addition, in the case of PC based OS, such as Windows, Intel MacOS, Linux (i386, x86_64), normal binary data reading order is little endian. Thus before processing our sample programs, user need to byte swaping process. In the case of bash, dd command can do such byte swap process as follows;
$dd if=(input binary) of=(output binary) conv=swab
Such byte swap does not need on native 64bit environments, such as Power PC, Intel Itanium (IA-64), SPARC, HP-UX, etc.

[Note] even though 64bit OS in intel/AMD CPU, such as Linux x86_64, MacOS, Windows 64bit, our sample programs for EXT (24000 pixels x 24000 lines) band data process as that;
a) A band data spliting into two files as Northern area and Southern area.
b) Then C and F90 program convert CCT to reflectances for each spliting files.
c) After conversion, marge two physical variable data into one file.
(please see the detail processes in shellscripts in tarball) This process is mainly due to the limitation of malloc (memory allocation).

Moreover, under the 32bit OS, due to the specification of 32bit OS (limitation of 32bit), it is impossible to generate more than 2 GB filesize file. If users try to analysis EXT data by 32bit OS, users have to treat EXT physical converted data as split files (e.g. Northern and Southern areas).

You can browse latest each band gridded data image in here. In addition, processed png images are also stored in ftp server. Please also download png images for target period as data convertion process check.

2. Data policy
2.1. Basic data policy to use data
CEReS H8/9 gridded data is limitted for non-commercial use. This data policy is same as JMA's one for academic-data-releasement. We also prohibit the distribution of our gridded data to third parties, except among the research communities.

2.2. Citation
Please cite our data as follows;
Himawari 8/9 gridded data are distributed by Center for Environmental Remote Sensing (CEReS), Chiba University, Japan.
In addtion, we have plan to submit our gridded program package methodologies and simple validation into scientific journal. After the acceptance in the journal, please cite published paper (will be listed here).

2.3. Quasi-realtime data
Due to data policy of JMA, quasi-realtime data (in the definition of JMA, realtime data is within 24 hours data since now) can not open without any restriction. Thus in the CEReS ftp server, we control the access for quasi-realtime data by IP address. If user who would like to access quasi-realtime data for non-commercial use only, please send e-mail with accessed global IP adress, and used objective to below e-mail address.
request4himawaridata _AT_ ceres.cr.chiba-u.ac.jp
We have no plan to change configuration on the CEReS ftp server by user registration. We will done via IP based restriction only.