Introduce of the Problems

Discuss about the Big Data Challenges in IOT and Cloud for Optimize Process.

One of the main features of utilizing IOT is to observing the behavior of many things to gain important optimize process, insights and so on. By utilizing the Big data several problems are solved such as store all the events, run analytical queries over the stored events and perform analytics (machine learning and data mining) over the data to gain insights (Cai et al., 2017). However there are main challenges to achieve the ability for performing real time analytics using big data such as correlate streaming events with my stored data in the operational database, process streaming events on the fly and store streaming events in the operational database. Rapid growth of IOT applications also presenting many challenges associate with the Big Data. Though lot of problems is likely to be solved but still needs sufficient research to migrate problems confronted. Data storage and management can raise challenges such as velocity, variety and volume of Big Data (Lee et al., 2013). Big Data velocity involves storage system to enable the capability to scale up quickly which is hard to gain with traditional data protection mechanism. However big data on the cloud is super expensive for utilizing a vast data volume. There are other certain challenges of Big Data in Data transmission process in several stages of lifecycle. Most of the challenges occur in the process of data collection, data integration and data management. Transferring large amount of data produces obvious risks in each of stages. It is also hard for traditional systems to efficiently analyze, manage and visualize unstructured data (Barnaghi, Sheth & Henson, 2013). The main reason for huge amount of data produced by IOT enabled devices is the growing number of internet-enabled devices utilized for several purposes by enterprises, individual and government. The large amount of data generated by IOT devices needs more computational power to process.  Consequently, data produced by IOT devices in several application domains are time perilous. Processing the vast amount of data in a timely manner is very demanding. As a result the whole process, plans and effective decisions are formulated for several applications sets.

This section provides the clear definition of the structure followed in this report. There is several section of this report for clearly describing the security and privacy regime in IOT. In every section there is proper heading with suitable subheading to demonstrate the broad about security and privacy regime associate with IOT. The demonstration of the followed structure is following-

• Executive Summary: Executive summary is the first part of the report where the summary of the whole report is presented. This section discusses about the Big Data challenges in the IOT. It helps to understand the whole report in short time. These sections also provide the brief summary of the whole content.
• Introduction: This part discuss about the basic challenges associate with the topic. It provides basic understanding of the chosen topic. It broadly describes the basic knowledge of big data and what are the challenges associates with the increment of data set. It also explains familiar application of big data associate with the IOT and cloud.
• Literature Review: this part of the project discuss about the published literature on the relevant topic. It provides a broad understanding of previous literature and the finding of that literature with reflection. This section provides a board review of existed research papers conducted through many sources such as Google scholar and e-libraries.
• Methodology: this is one of the crucial parts of the project where the suitable method will be chosen for completing the report. After reviewing and compare past and previous methodologies my own perspectives are illustrated. This section illustrates five methodologies which are utilized to propose the desired knowledge of big data challenges associates with the IOT and cloud.
• Conclusion: in this part the result of the whole report is illustrated. This part widely discusses about the big data challenges in the IOT and how to measure them in term of efficiency, feasibility and connectivity.

Structure of the report

Milestone 3

Introduction

Nowadays, there are several modifications are happening in the technology field especially in the platform of cloud, big data and internet of things. The internet and the amount of the users are growing rapidly, it also producing vast amount of data day by day. According to some research (Bessis & Dobre, 2014). Everyday more than 2.35 quintillion bytes are produced. The vast amount of data is produced by the user who is creating this for utilizing many applications. It is getting hard to manage this vast amount of data which are being produces for several uses. The collaboration of machine and human is producing many challenges as well. According to Chen, Mao & Liu, 2014, most of the devices or system used by user in a daily basis produces vast amount of data which are always not visible as hidden information stored in the devices. Internet of things is triggering the database management growth in future. The challenges occur to manage large dataset from every system or devices. Prediction capabilities of the applications and services and higher level of decision making are very essential to achieve the full advantage from the services and context aware data intensive applications and make the important information or valuable transparent and available at a much higher frequency. As per Chen et al., 2015, big data has become very essential to turn the vast amount of data into information that can be transferred with a high frequency. Big data can be define as high velocity, volume an verity information which enable innovative way of information processing and to gain enhanced insights through automation process, effective decision making and analytical tool. According to research paper from Gartner, almost every business, enterprise and individuals are producing vast amount of data which can be analyzed through automation process to manage it more effectively (Lee & Lee, 2015). In fact the life of data can be very short and this may become obsolete after some time. So, efficient usage of Big Data analytics results will bring good useful insights from high volume, variety of data. Sometimes quality of the data is a concern area because it fetches data from different applications for making decisions. Not all the data captured from various devices are useful for making decisions (Li, Da Xu & Zhao, 2015). These are just information and the information has to be converted into knowledge for decision making.

Literature Review

Fan & Bifet, 2013, state that for the large and complex volume of big data it is very tough to manage and analyze the data. Effective data management would not be possible by utilize typical Rational dataset management system (RDBMS). For the vast amount of big data it is also very hard to extract proper information in an effective manner. Generally, the raw big data is the source of the desired information. Data is consisted with each other as raw input. Individual’s data has no value. Vast amount of data provide meaningful information with several patterns and trends.

There are several research papers available which are available in the web. Some of the research tried to estimate all the challenges could possible occurred by the overwhelming amount of data (Jin et al., 2014). Technology shifts more data through various applications like wireless sensors, smart devices, social media etc. This paper focuses on the improvement the performance of the old services and offer new services in an open and dynamic environment. 

Big data is the specific type of data set, which is extremely complex as well as voluminous and the traditional data processing application software is not at all sufficient for dealing with these complex data sets (Gudivada, Baeza-Yates & Raghavan, 2015). Big data is one of the most important and significant data sets that have the advantage to deal with any type of data and there is no restriction to data size or data limit. It can be easily referred to as the evolving term, which describes all types of voluminous amount of unstructured, structured and semi structured data, which comprises of the string potential for information mining. This particular type of data, big data is subsequently categorized by the help of 3Vs (Mishra, Lin & Chang, 2015). These 3Vs are highest volume of the data, broader variety of data and finally the highest velocity of data. The data that is being processed is checked with the help of this big data. There is no specific data volume for big data and thus this term is utilized for describing the petabytes, terabytes and finally exabytes that is captured over time. This specific type of data inundates any organization on the day to day basis. The big data could be easily analyzed for the insights, which eventually lead to the better business decisions and also strategic business movements.

Big data eventually refers to the proper utilization of user behavior analytics, predictive analytics and any other method of advanced data analytics. These advanced data analytics are solely responsible for extracting significant value from the data and also to the specific data set size (Obitko, Jirkovský & Bezdí?ek, 2013). The proper analysis of the data sets could easily find out various newer correlations that help in spotting the business trends, preventing the diseases and combating the crime and many more.

Present and Past Work

As per Hassanalieragh et al., 2015, for the description of the 3Vs, that is volume, velocity and variety, big data has become explicitly popular and well accepted by every business or company. The organizations or businesses eventually collect the data from a vast variety of various sources. These sources majorly include all types of business transactions, social media and finally sensor information or data and data related to machine to machine. In the previous days, storage of the volumetric data was extremely difficult, however, with the rise of various technologies like Hadoop, the complexities or problems are solely reduced to a greater extent (Sivarajah et al., 2017). The next V amongst the 3Vs is velocity. Data streams in at an unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors and smart metering are driving the need to deal with torrents of data in near-real time. This is extremely important for any organization or business as it increases the velocity or speed of the business processes or operations. The final V is the variety. It eventually refers to the fact that any type of data comes in every format. The various formats of data are structured, numeric and unstructured (Gubbi et al., 2013). The traditional databases are not affected in this sector and hence the emails, audio, video and various financial transactions are checked in the process.

Apart from the above mentioned 3Vs two more dimensions are present within the big data. These are variability as well as complexity. Variability refers to the fact that big data does not comprise of any type of consistency, which means the data is not at all fixed (Hashem et al., 2015). There is always a high chance of variation or changeability. Thus, easy adaption and data change is possible in big data. Moreover, this big data is much complex to crack and hence there is extremely less chance that the complexity is almost zero in this particular data type.

Internet of things can be defined as the significant network of various physical devices, home appliances, vehicles and several other items that are eventually embedded with the software, electronics, actuators, connectivity and sensors that solely enable all the objects in proper connection as well as exchanging of confidential information and data (Yang, 2014). All these things that are related to internet of things and cloud computing could be uniquely recognized by the specific embedded computing system and has the ability in inter operability in the existing infrastructure of the Internet connection. This internet of things enables all the objects that are to be controlled as well as sensed remotely within the existing network infrastructure. This helps to create various opportunities to directly integrate the typical physical world within the systems that are completely based on computers. This helps to improve the efficiency, to provide economic advantages and finally to provide accuracy of the systems. This eventually reduces the efforts as well as intervention of the human beings (Strohbach et al., 2015). When IoT or cloud is augmented with sensors and actuators, the technology becomes an instance of the more general class of cyber-physical systems, which also encompasses technologies such as smart grids, virtual power plants, smart homes, intelligent transportation and smart cities.

Big Data Definition

Big data and Internet of things or cloud computing are both extremely important and significant technologies in the technological world. When these two popular and famous technologies are amalgamated together, they bring out one of the best technologies for the world. The exclusive growth of the number of various devices that are connected to the technology of IoT or Internet of Things as well as the exponential increment of consumption of data has eventually reflected on the fact that the growth or development of big data can easily overlap the technology of IoT or Internet of Things (Wortmann & Flüchter, 2015). The entire management of the big data within the continuous expansion of network has solely given rise to any non trivial concern about the efficiency of data collection, data security, and analytics of data and processing of data. For the purpose of addressing all of these problems or concerns, the researchers have subsequently examined each and every challenge that is linked with the proper deployment of Internet of things. Although, there are various studies on the big data and Internet of things, there is a strong convergence of all of these areas and thus creating various opportunities to flourish the big data or analytics for the systems of internet of things and cloud computing. 

Big data comprise of various issues or challenges when it is mixed with the Internet of things and cloud. The various challenges of big data with their probable solutions are given below:

1. i) Heterogeneity and Incompleteness: The first and the foremost challenge of the big data when present in the technology of Internet of things and cloud is heterogeneity and incompleteness (Zaslavsky, Perera & Georgakopoulos, 2013). The difficulties of big data analysis derive from its large scale as well as the presence of mixed data based on different patterns or rules or heterogeneous mixture data in the collected and stored data. In the case of complicated heterogeneous mixture data, the data has several patterns and rules and the properties of the patterns vary greatly. Moreover, there are various uncertainties of data that data might lose the integrity, while it is being managed. All these occur when big data is working with internet of things (Zanella et al., 2014). The solution to the above mentioned problem is that the data should be collected or stored properly for reducing the problem of heterogeneity and the issue of data incompleteness is solved by my qualitative data analysis.
2. ii) Working with Large Set of Data: There is a challenge in big data that it cannot work with smaller data sets and thus only huge amount of data sets are involved in the process. Often the user suffers from various issues due to this as he is not being able to link internet of things with big data if the data set is smaller. For solving this problem, various approaches are to be taken by the user

iii) Scale as well as Complexity: The third significant challenge of the big data in IoT and cloud is that it is extremely complex and the management of the larger and increasing data volume is considered as the most challenging issue ever for any user. Thus, the user needs to manage or control the data properly and perfectly. The solution to the above mentioned problem is that the data should be managed with the help of various software tools.

1. iv) Data Clustering: Another significant issue with the big data in internet of things is data clustering. This data clustering refers to the task of clustering or grouping the specific object set in the particular method by which the objects within the same cluster work together (Da Xu, He & Li, 2014). This becomes a major problem for Internet of things and cloud computing as it cannot work with data clustering. The solution of this challenge is to store the data without being clustered.

This is an Open source mission which is mainly managed by the Apache Software Foundation. By using this Hadoop it is easy to collect and handle the big data. This is proposed for the purpose of parallelizing the processing’s of the data and this is done by computations of the nodes in order hurry the computations and also to hide the latency (Perera et al., 2014). The Hadoop mainly consists of two components and this includes the Hadoop Distributed File system and the Map Reduce engine. The Hadoop Distribution File System is associated with storing the enormous data in a constant way and is also associated with reproducing it to the user applications which are at high bandwidth. The MapReduce can be considered as a framework which is used for eth purpose of process the data sets which are massive and are present in a distributed fashion through numerous machines. 

This is mainly constructed in the form of board programming paradigm. It is seen that some of the original employments are offered all the key needs related to the parallel execution, fault tolerance, balancing of load, and manipulation of data. This has been named as this mainly includes two kinds of abilities from the functional computer languages which are existing and this includes the map and reduces (Stankovic, 2014). The MapReduce framework is associated with gathering all the sets which are consisting of common keys from all the records which is followed by joining them together. This initially results in the formation of one group for each of the different keys which are produced. This is one of the new technologies, which is just an algorithm and a technique by which all the data can be fitted. In order to obtain the best from the Map Reduce there is just a need of an algorithm. For this there is a need of collection of products and the technologies which are created in order to manage all the challenges faced by the big data.

This is the database model which is present inside the Hadoop framework and looks like the original system of the big table. The HBase also consist of a column which is associated with operating as the key and is the only index which can be used for the purpose of getting back to the row (Cui, 2016). The data which is present in the HBase can also be saved as various sets, and in this the subjects which are present in the non-key columns is represented by the values.

The tools which has already been deployed for data warehousing is not at all suitable and especially in the situation where it is seen that the data is accessible everywhere. Besides this they are also costly and are often operated privately. Sometimes the Hive may also be thought as the necessary portion of Hadoop system and is viewed at the top that principally is the organization for the data warehouse (Al-Fuqaha et al., 2015). Hive cannot treat with applications and transactions of the real time those are achieved online. The motivation behind it is a complicated technique.

There are various advantages of benefits of big data within internet of things. The several benefits are given below:

1. i) Detection of errors: The first and the foremost advantage of big data in internet of things is that it helps to detect any type of error instantly and hence the users are benefitted from the technology instantly (Chandrakanth et al., 2014). Moreover, the operations or processes of the business are improved eventually since the error detection is done easily.
2. ii) Improved Strategies: The second significant benefit of big data in internet of things is that it helps in improving all the new strategies within the business properly and thus the business enjoys major advantages and benefits from these technologies. The main reason of this the presence of data analytics.

iii) Cost Savings: Another important benefit of the big data in internet of things is that it is extremely cost effective and does not incur huge cost for the user (Mukhopadhyay & Suryadevara, 2014).

1. iv) In Memory Databases: The fourth benefit of big data in internet of things is that it comprises of an in memory database and hence other databases are not required.

The various demerits of big data within internet of things are given below:

1. i) Requirement of Special Computer Power: Big data in internet of things needs special computer power and thus is termed as one of the most difficult technologies.
2. ii) Utilizes Real Time Insights: The next demerit is that utilizes real time insights and hence the user suffers from major issues (Biswas & Giaffreda, 2014).

There are some of the future directions of big data in Internet of things. They are as follows:

1. i) Increasing Data Volume: The data volume will be growing as days will pass and thus data management issue would be reduced (Whitmore, Agarwal & Da Xu, 2015).
2. ii) Tools for Analysis: Various other tools would analyze the data properly and thus data loss would be highly reduced.

iii) Machine Learning: This technology would be mixing up with machine learning and hence the technology would be enhanced.

Conclusion

Therefore, from the above research report, conclusion can be drawn that big data, Internet of things and cloud computing are the most promising technologies in the modern world. These two technologies help in analysis of data. Big data deals with various larger sets of data and makes the data operations much easier for the users. big data not only means vast volume, as there is three type such as variety (types of content), velocity (rate of the data production and processing rate) and volume (number and size). The volume expedition is been a one of the most dominant issue in the IOT industry. The volume rate is increased from terabyte to petabyte and Exabyte which is equal to one million Terabyte.  Where variety refers to the type of content such as data format, under structured, semi structure which may produce from different type of sources such as devices, machines, sensors many more. The Speed of the production of data and to process to the data to generate valuable insights referred as Velocity. These types of data sets are subsequently analyzed computationally for the purpose of revealing the trends, patterns and associations. This is specially related to the human interactions or behavior. The three types of data that are structured, semi structured and unstructured are well managed in this particular technology and increasing the significant potential of information mining. Internet of things refers to the particular inter connection via the Internet of computing devices embedded in everyday objects, enabling them to send and receive data. It is considered as the best technology and thus is much popular in the entire technological world. Internet of things is also termed the data transfer that is done on the Internet without needing the human to computer or human to human interaction. The above research report has clearly outlined various important and significant features of the amalgamation of internet of things, cloud computing and big data. The several advantages, disadvantages and future directions of the cloud computing are mentioned here with relevant details. 

References

Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., & Ayyash, M. (2015). Internet of things: A survey on enabling technologies, protocols, and applications. IEEE Communications Surveys & Tutorials, 17(4), 2347-2376.

 Barnaghi, P., Sheth, A., & Henson, C. (2013). From data to actionable knowledge: Big data challenges in the web of things [Guest Editors’ Introduction]. IEEE Intelligent Systems, 28(6), 6-11.

Bessis, N., & Dobre, C. (Eds.). (2014). Big data and internet of things: a roadmap for smart environments (Vol. 546). Springer International Publishing.

Biswas, A. R., & Giaffreda, R. (2014, March). IoT and cloud convergence: Opportunities and challenges. In Internet of Things (WF-IoT), 2014 IEEE World Forum on (pp. 375-376). IEEE.

Cai, H., Xu, B., Jiang, L., & Vasilakos, A. V. (2017). IoT-based big data storage systems in cloud computing: perspectives and challenges. IEEE Internet of Things Journal, 4(1), 75-87.

Chandrakanth, S., Venkatesh, K., Uma Mahesh, J., & Naganjaneyulu, K. V. (2014). Internet of things. International Journal of Innovations & Advancement in Computer Science, 3(8), 16-20.

Chen, F., Deng, P., Wan, J., Zhang, D., Vasilakos, A. V., & Rong, X. (2015). Data mining for the internet of things: literature review and challenges. International Journal of Distributed Sensor Networks, 11(8), 431047.

Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile networks and applications, 19(2), 171-209.

Chen, M., Mao, S., Zhang, Y., & Leung, V. C. (2014). Big data: related technologies, challenges and future prospects (p. 35). Heidelberg: Springer.

Cui, X. (2016). The internet of things. In Ethical Ripples of Creativity and Innovation (pp. 61-68). Palgrave Macmillan, London.

Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions on industrial informatics, 10(4), 2233-2243.

Fan, W., & Bifet, A. (2013). Mining big data: current status, and forecast to the future. ACM sIGKDD Explorations Newsletter, 14(2), 1-5.

Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of Things (IoT): A vision, architectural elements, and future directions. Future generation computer systems, 29(7), 1645-1660.

Gudivada, V. N., Baeza-Yates, R. A., & Raghavan, V. V. (2015). Big Data: Promises and Problems. IEEE Computer, 48(3), 20-23.

Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Khan, S. U. (2015). The rise of “big data” on cloud computing: Review and open research issues. Information Systems, 47, 98-115.

Hassanalieragh, M., Page, A., Soyata, T., Sharma, G., Aktas, M., Mateos, G., … & Andreescu, S. (2015, June). Health monitoring and management using Internet-of-Things (IoT) sensing with cloud-based processing: Opportunities and challenges. In Services Computing (SCC), 2015 IEEE International Conference on (pp. 285-292). IEEE.

Jin, J., Gubbi, J., Marusic, S., & Palaniswami, M. (2014). An information framework for creating a smart city through internet of things. IEEE Internet of Things Journal, 1(2), 112-121.

Lee, G. M., Crespi, N., Choi, J. K., & Boussard, M. (2013). Internet of things. In Evolution of Telecommunication Services(pp. 257-282). Springer, Berlin, Heidelberg.

Lee, I., & Lee, K. (2015). The Internet of Things (IoT): Applications, investments, and challenges for enterprises. Business Horizons, 58(4), 431-440.

Li, S., Da Xu, L., & Zhao, S. (2015). The internet of things: a survey. Information Systems Frontiers, 17(2), 243-259.

Mishra, N., Lin, C. C., & Chang, H. T. (2015). A cognitive adopted framework for IoT big-data management and knowledge discovery prospective. International Journal of Distributed Sensor Networks, 11(10), 718390.

Mukhopadhyay, S. C., & Suryadevara, N. K. (2014). Internet of things: Challenges and opportunities. In Internet of Things (pp. 1-17). Springer, Cham.

Obitko, M., Jirkovský, V., & Bezdí?ek, J. (2013, August). Big data challenges in industrial automation. In International Conference on Industrial Applications of Holonic and Multi-Agent Systems (pp. 305-316). Springer, Berlin, Heidelberg.

Perera, C., Zaslavsky, A., Christen, P., & Georgakopoulos, D. (2014). Context aware computing for the internet of things: A survey. IEEE communications surveys & tutorials, 16(1), 414-454.

Sivarajah, U., Kamal, M. M., Irani, Z., & Weerakkody, V. (2017). Critical analysis of Big Data challenges and analytical methods. Journal of Business Research, 70, 263-286.

Stankovic, J. A. (2014). Research directions for the internet of things. IEEE Internet of Things Journal, 1(1), 3-9.

Strohbach, M., Ziekow, H., Gazis, V., & Akiva, N. (2015). Towards a big data analytics framework for IoT and smart city applications. In Modeling and processing for next-generation big-data technologies (pp. 257-282). Springer, Cham.

Whitmore, A., Agarwal, A., & Da Xu, L. (2015). The Internet of Things—A survey of topics and trends. Information Systems Frontiers, 17(2), 261-274.

Wortmann, F., & Flüchter, K. (2015). Internet of things. Business & Information Systems Engineering, 57(3), 221-224.

Yang, S. H. (2014). Internet of things. In Wireless Sensor Networks (pp. 247-261). Springer, London.

Zanella, A., Bui, N., Castellani, A., Vangelista, L., & Zorzi, M. (2014). Internet of things for smart cities. IEEE Internet of Things journal, 1(1), 22-32.

Zaslavsky, A., Perera, C., & Georgakopoulos, D. (2013). Sensing as a service and big data. arXiv preprint arXiv:1301.0159.