INFORMATION TECHNOLOGY DIVISION

Indian Census has always been in the fore front in adopting new technology for data processing as well as dissemination and other related activities. Adding another feather in its cap, for the first time in the history of Indian Census, it is decided to collect the data through digital mode in the 2021 round of Census. The enumerators, of course, would be at liberty to choose to collect data either through their own smart phone using the mobile applications developed by ORGI or they can opt for conventional paper questionnaire as well.

The broad outline of the methodology is as follows:

Data Collection will be done by any one of following Modes:- :
• Mobile Applications
• Self Enumeration
(For mobile applications and self-enumeration, Data will be transferred automatically by the System to any one of 3 Data Centers of ORGI located at Delhi, Lucknow and Bengaluru.)
• Paper questionnaires –will be scanned, digitized through ICR at 18 Data Capture Centers of ORGI then sent to any of 3 data Centers as mentioned above.

Management & Monitoring through Census web portal (CMMS): A unique initiative has been envisioned and put in place, to manage and monitor all the preparatory and on-going activities of Census Operations through a web portal on real time basis. Starting from maintaining the location particulars of all the administrative areas of the Country, CMMS would also be a repository of information on the appointment of various census functionaries, training activities, work allocation, logistics, real-time progress of field work, online data entry of consolidated abstracts etc., just to name a few.

Data Security:

Ample measures have been put in place to ensure data security at every stage of the data cycle, starting from data collection, during transmission and while the data is stored inside the ORGI server. Some of the measures to ensure data security are:- :

• Data collection is done only by well-identified user.
• Multiple Level Layered Security approach while data is on mobile, in transmission or at rest.
• Data collected will be suitably encrypted before saving in mobile with date & time stamping.
• Once Data is transmitted to ORGI Server, Data will be removed from mobile after a due date automatically.
• All the data accessed along with logs and security features will be maintained meticulously.
• Data in ORGI servers will be stored in encrypted form only. The encrypted data will be decrypted in Memory for further processing and once the operation on data is over, it will not be available in decrypted form to the users.
• Data Backup/Disaster Recovery – All Server and Storage receiving/processing or keeping the data in safe mode are placed at 3 Data Centres (Delhi/Bengaluru /Lucknow) to protect the data from any un-foreseen situation.
• To protect ORGI Data Centres and its data from volumetric attack, Network intrusion, SQL injection, Cross Scripting, data leakage, unauthorised data access etc., ORGI has planned to setup the Captive Security Operation Centre (SOC) and Security Incident Event Management (SIEM) with suitable HW/SW appliance at all the 3 ORGI Data Centres.

Technology Options for Census 2011 IT resources were created at 18 data centres by installing 36 Nos Microsoft Windows 2008R2 servers, SQL Server -2008, Windows 7 Professional Clients, High speed heavy duty duplex scanners (Kodak) and backup HP Storage Works EVA6400, Capacity – 10/100 TB. In addition to above Hardware & Software an Intelligent Character Recognition (ICR) software eFlow4.5 has also been installed to meet the challenges of Census-2011. About 1200 technical officers and about 500 contractual operators were engaged for electronically data capturing at 18 data centres and processing of census data at DP Division, New Delhi.The use of ICR technology after scanning the schedules and use of Computer Assisted Coding (CAC) saves a lot of Government expenditure incurred in the past on setting up of Regional Tabulation Offices. The services of a System Integrator were utilized at 18 Data Centres for scanning operations and data file creation. All the required software & MIS tools are developed by the officers of Data Processing Division, ORGI, New Delhi.

Following Server Hardware and storage has been installed in D.P. Division and in 18 Nos Data Centers in India for data processing activities- :
• HP ProLiant DL380G6 Quad Core Base Servers
• HP Storage Works EVA6400, Capacity – 10/100 TB
• Kodak High Speed Scanners

At 2001 census, large scale hardware up-gradation had been made at 15 Data Centres and in DP Division, New Delhi. This was required for using the latest “Automatic Form Processing Technology" using “Intelligent Character Recognition (ICR)” technology. The OMR/OCR/ICR technologies were evaluated and ICR was found to be acceptable being the viable solution for census data processing activities. IT resources were created at 15 data centres by installing 45 NT servers, 1060 PIII PCs, 25 High speed heavy duty duplex scanners (Kodak) and used backup devices like ZIP SLR & DLT drives. In addition to above Hardware & Software, an Intelligent Character Recognition (ICR) software eFlow has also been introduced and installed for the first time to meet the challenges of Census-2001. About 1200 technical officers and about 500 contractual operators were engaged for electronically data capturing at 15 data centres and processing of census data at DP Division, HQr., ORGI, New Delhi. The paper schedules were transported to the nearest allotted data center (resulted in reduced transportation cost) for scanning and data file creation activities. The scanning activities were carried out on 24x7x30 basis. The technology enabled processing of cent percent data, that is, more than one billion records (228 million paper forms were scanned) for the first time in census history. The scanned images were archived for permanent storage. Image enabled Computer Assisted Coding (CAC) was applied for certain fields. The use of ICR technology after scanning the schedules and use of CAC (eliminated setting up of Regional Tabulation Centres for manual coding) saved the Government lot of expenses incurred in the past on setting up of Regional Tabulation Offices. The services of a System Integrator were utilised at 15 Data Centres for scanning operations and data file creation. The use of technology has helped to enhance the skills of EDP officers of ORGI. The DP Division officers have developed the data processing software in-house.

The DP Division has assisted various Technical Divisions of ORGI for data processing projects, such as:
• Preparation of District Hand Book ( Social Studies Division)
• Large size village study data entry and processing (Social Studies Division)
• Extraction of data from 2001 census data and tabulation for slum project (Census Division)
• Scanning and ICR based processing for Verbal Autopsy (Vital Statistics Division)
• Scanning and ICR based processing for 5th Economic Census of Central Statistical Organisation.

During 1991 Census, a drastic change in the data processing activities in ORGI was taken place. ORGI had setup its own computing facility by installing Medha-930 main frame system at DP Division, Pushpa Bhawan. Dumb terminals, under Unix operating system, connected to the servers at 15 data centres, were used for data entry. Data movement between various data centers and Computer centre at Delhi was done through magnetic tapes. The data schedules were coded at various regional tabulation centres (about 163) throughout India before sending the schedules to the allotted one of the 15 data centres for data entry. Master data files creation, data editing on basic fields and lower level tabulations were processed at 4 Regional Processing Centres namely Delhi, Bhopal (Madhya Pradesh), Bhubaneshwar (Orissa) and Chennai (Tamil Nadu). Major editing considering all the fields, processing and generation of all the tables at various levels were taken up in DP Division, ORGI, HQr. All the required software (for data validation, editing, processing and tabulation) were developed by the officers of Data Processing Division, ORGI, HQr. First time, in the 1991 census, DP Division generated the camera ready copies of the tabulations in Hindi as well as in English for publication. The cent percent data was captured. The processing was done on the cent percent data of workers & SC/ST. Only 10 per cent records were processed for other tabulations. Around 1200 ORGI officers including officers at 15 data centres were engaged for the entire census data processing activities. The software were developed in-house by DP Division officers.

In 1981 Census, first time in Indian census, the data entry activities were decentralised and 15 data centres were setup in major states including one at headquarters, ORGI. Each centre was allotted more than one state (adjoining states) to convert paper-based information into machine-readable form using “key to disk” machines provided by GCS, ECIL & ICT. The data processing was done using HP1000, CD-Cyber 730 & NEC-1000 computer systems at NIC, New Delhi and Regional Computer Centre (RCC), Chandigarh. The in-house data processing facilities were not available in ORGI. The data schedules were coded at various regional tabulation centres throughout India before sending the schedules for data entry to allotted one of the 15 data centres. Around 1200 ORGI officers including officers at 15 data centres, were engaged for the entire census data processing activities. All the required software (for data validation, editing, processing and tabulation) were developed by the officers of Data Processing Division, ORGI, Headquarters.

Census 1971 was the 11th in the continuous series of censuses since 1872. It is well-known that the processing of the census data occupies a very important place in the census. The census enumeration may be thorough and accurate but the census tabulation will not be accurate and useful unless the raw data are properly processed. There are various methods of data processing and the choice of a suitable method depends upon the circumstances of the country. In 1971 census, most of the tables were prepared from a 10 percent sample of the rural population and a 20 percent sample of the urban population, with the help of electronic computers, mainly IBM-1401s.

In 1961 census, a modest beginning was made with the adoption of “Unit record” machines. Prior to 1961 censuses, the data collection, data entry and processing were done manually. In 1961 census, the Hand Punching machines (inserting one card at a time) using 80 column (Hollerith) punch cards were used for converting data into machine readable form. The processing was done on the sample (data) selected from the entire data. Around 70 ORGI officers were involved in the data entry, programming & machine operations. The data schedules were coded at various regional tabulation centres throughout India before sending the schedules for data entry. Reproducer was used to duplicate the entire set of punched cards. Data Processing was done by using Verifiers, sorter machines. “Serial Rolling Total Tabulator cum printer (SRTT)” was used for tabulation and printing of census tables. Data backup system was not available at that time.