Big Data

Kate Osborn • Mar 30, 2012
Big Data - the next Big thing..?

So what is Big Data?

Every day an increasing amount of information is gathered and stored by all manner of websites, companies, organisations and products. We record data on everything from weather stations to shopping habits, social media to traffic trends. The volume of this data is growing at an alarming rate as we develop more complex programs for recording data down to the smallest detail for analysis, reporting and business intelligence, and the type of data that we can record is also evolving. 

Every day, we create 2.5 quintillion bytes of data

The growing need for "trending" means that not only does an individual customers details now need to be stored, data for all customers is now required to be accessed quickly and analysed as a whole, to find trends and patterns across the whole collection of data whether that be thousands, millions or trillions of values to be analysed

90% of the data in the world today has been created in the last two years alone

Structured/UnStructured Data
Data now also does not only mean values and figures that can be represented in spread sheets and graphs as previously favoured. "Data" can now be stored as images, documents, emails, text messages, audio, video, etc.. These various data types are commonly known as “non-structured” or “unstructured”, ie: unable to be stored and analysed in a typical relational database. This is the real issue faced by todays companies, the data they have stored is not able to be analysed in the traditional way, but simply ignoring this data would be foolhardy

Many organizations are becoming overwhelmed with the volumes of unstructured information — audio, video, graphics, social media messages — that falls outside the purview of their “traditional” databases. Organizations that do get their arms around this data will gain significant competitive edge

For example in the past medical records would be held in vast paper files containing Xrays, reports, doctors notes, etc.. Now those Xrays are recorded electronically and can be accessed from a doctors PC without the need for any physical document store. One hospital will store and record thousands and thousands of Xrays, one hospital group will have millions. Big Data is the means of retrieving this information easily and comparing and analysing this data in a way which has never been possible previously 

“We are only looking at what we have in our data warehouses, it’s not going to be enough for us to get the insights that we need. If you’re a retailer and you were not using all the information you could to judge your customers’ buying patterns, then the retailer across the street probably will, and they’ll steal your customers. That’s the realization, I think, that drove a lot of people to think that they should be capturing much, much more” 

Data Timeline
The other demand for data is timeliness. The demand for “real-time” data is increasing, data that is 24 hours or even 1 hour out of date could be the difference between a successful business decision. The accuracy of the information received often depends on the age of the data analysed, for example a hotel needs up to the minute room allocation data in order to receive new customers. If this data was 10 minutes old then a room could be allocated twice, or customers turned away when a room is actually available. For Credit Card companies the ability to recognise fraudulent activity quickly and stop the stolen card saves them many thousands of pounds
Real-time analytics is the big demand. The Holy Grail is “getting and making effective use of information as it happens.” Whoever can crack this will be the forerunner

'Data delayed' is 'Data Denied' 

Velocity – Often time-sensitive, big data must be used as it is streaming in to the enterprise in order to maximize its value to the business 

So now we understand that “Big Data” is the means of accessing and analysing large quantities of data regardless of size, format or timestamp

The next question is how do you compare millions of images to find trends and patterns? Originally in a "structured" database this would not be possible (useable), as a relational database is designed to analyse data, not images. For the latter we need a platform to query "non-structured" or “semi-structured” data. New technologies are now emerging for the storing and analysing of this large volume structured and non-structured data, but it is still early days

Platform:
NoSql (short for Not Only SQL) is the term used to describe non-relational databases that can handle both structured and unstructured data. There are several contenders for big data databases, some use no SQL at all while others utilise certain areas but avoid joins, etc..

Contenders:
Apache Cassandra is the most common NoSQL database, as used by Facebook
Apache Hadoop (to be integrated with SQL Server 2012 – more on this in the next blog)
SimpleDB
Google BigTable
MapReduce
MemcacheDB
Voldemort

It has been predicted that “Organisations that will leverage the new data types will outperform their peers by 20% by 2015” - If you ignore big data, your competition will not

Once Big Data takes off the possibilities opened up will create a lot of changes. New cars fitted with “smart boxes” recording data about your driving could be accessed by Insurance companies, with lower premiums offered to those drivers who can prove they are a safer risk. Medical data compared for millions of patients to identify high risk groups for certain illnesses or diseases, and tests developed to pick up the warning signs earlier for those most at risk. GPS data analysed to predict traffic volume and flow at peak times, Event data built into our satellite navigation to warn of possible delays. Live streaming from traffic cameras to our iPhones’. Facebook photos used to identify benefit fraudsters

The realization that time to information is critical to extract value from data sources that include mobile devices, RFID, the web and a growing list of automated sensory technologies
by Garry Lowther 02 Apr, 2024
The newest version of TriSys Mobile is now available for Apple and Android users
Top and Bottom Grid Paging now also available in TriSys Apex
by Francisco Janes 18 Mar, 2024
Top and Bottom Grid Paging now also available in TriSys Apex
by Francisco Janes 13 Mar, 2024
TriSys Apex now features editable contact previews designed to streamline your cloud recruitment database experience.
by Francisco Janes 12 Mar, 2024
TriSys Cloud Remote desktop for recruitment professionals. Cloud hosted recruitment software.
by Garry Lowther 05 Feb, 2024
We are planning some very exciting innovations and enhancements to the products and services of TriSys Business Software in 2024: TriSys Apex 2024 Our cross-platform web browser recruitment CRM was released in 2014 and uses the TriSys cloud platform to process all its data and functionality. Customers with any type of computer, tablet or phone can access their recruitment data and conduct sophisticated workflows to drive their business forward, wherever they are and at any time. We plan a number of innovations this year, including the port of the temps booking sheet, and integration of AI. TriSys 2024 The latest version of our premiere windows desktop recruitment CRM is now available to all existing customers. This application was first released in 1992 and is a rock-solid enterprise grade recruitment business app. For customers on the 2023 version, the application will automatically upgrade. All new features will be documented in the coming months. Our SDK is still the only recruiting software toolkit for Microsoft .Net. TriSys for the Microsoft Store was delivered using Visual Studio 2022. TriSys Cloud Platform Our cloud infrastructure platform was conceived and built in 2002. It has grown and matured to comprise literally hundreds of cloud servers distributed across the planet. Customers use our platform to store and process their large corporate data assets, via our software products and services. Expect more integration with world class web services, including artificial intelligence, through our Web API . Third Party Integrations The TriSys Recruitment Software Platform enables unlimited integration flexibility including standard and complex integrations across multiple popular third party industry suppliers enabling you to use TriSys as a process hub offering a single source of truth. We have recently built a sophisticated integration with Zapier , and this year we will fully integrate with additional best-of-breed third party business platforms including artificial intelligence (AI). TriSys Mobile Suite Recruiters can access their recruitment database from their tablet and phone by installing our app , however the mobile landscape is changing with a new class of 'app-builders' utilising low-code or no-code technologies. Due to the power and flexibility of the TriSys Web API, Recruitment agencies can cost-effectively commission their own dedicated and branded mobile app to engage with their clients and candidates. A new truly-native version of TriSys mobile will be released to the stores in Q1 2024. TriSys Web API Almost all our recent innovations centre around our web application programming interface which provides a fully secure set of programmable end-points not just into the TriSys Cloud Platform, but also to a host of third party global ecosystems, to provide customers with unlimited flexibility to deploy a range of sophisticated recruiting business benefits. Web API plans for 2024 are to add mobile app-specific functionality allowing the TriSys platform to reach a wider range of customers by utilising AI. TriSys Platformix Five years ago, we introduced a website builde r technology utilising a mix of cloud platforms, which we called Platformix . This is the successor to Web Jobs in that it is a full recruitment website content management system (CMS) with very tight integration with the TriSys recruiting database, plus state of the art blogging, SEO compliancy, and analytics. We will continue to invest in building TriSys Widgets to allow our customers to build high engagement web portals. We will also continue integration with Stripe to allow Gig-Economy services to be built for our customers. TriSys Open Source We now have full GitHub source code integration with all our software build tools, meaning that TriSys is now the only commercial recruitment software vendor to offer an open-source model. From 1st January 2024, existing customers are able to license TriSys' source code to build their own highly-specific recruitment CRM app for desktop, mobile and web. We are also planning to partner with other organisations to build complex line of business applications for all industries, based upon the thousands of man-years development and testing which has been invested in the TriSys Platform code base. TriSys Remote Desktop The original TriSys ASP was launched 22 years ago , and comprises a windows hosted desktop running on our cloud. Last year we launched a Windows 11 version with full compatibility with Microsoft Office 365 allowing customers to have full control over the apps they can install. This year we will continue to support all Microsoft supported versions of Windows.
by Francisco Janes 28 Nov, 2023
Announcing Real-Time E-Mail Synchronisation for your Business
by Francisco Janes 24 Nov, 2023
Announcing a New TriSys Integration: Daxtra Search Nexus and Capture
by Francisco Janes 23 Nov, 2023
Enhancing Recruitment Efficiency with ChatGPT
by Garry Lowther 22 Nov, 2023
This afternoon, I researched how to construct intrinsic load balancing within our existing production Web API to take advantage of multiple Web API servers on the TriSys Cloud Platform . I decided to use ChatGPT, given the extremely high profile of OpenAI's flagship service after the amazing events over these last few days. The dialogue between myself and ChatGPT was akin to me working with another ASP.Net Web API expert who knew more about specific topics than I did, however it was necessary for me to test the generated code and indeed suggest a fix to an issue which prevented the code working in a live environment. Me: In ASP.NET Web API, how can I route a request to another Web API server? ChatGPT: In ASP.NET Web API, routing a request from one Web API server to another is a task that typically involves creating a proxy or a forwarding mechanism within your API. This is useful in scenarios like load balancing, microservices architecture, or when integrating with external APIs. Here's a basic approach to achieve this: Me: With regards to Handling the Response and your advice: Once you receive the response from the target server, you can either directly return this response to the client or process/modify it as necessary. Can you supply .NET code which demonstrates this? ChatGPT: After numerous exchanges like this over 20 minutes, and much copy and pasting, I had working code running in the Visual Studio debugger which was remoted to allow my Postman session to make Web API requests. Eventually, we arrived at a working solution. Obviously I asked ChatGPT to summarise this work for this blog post: Title: Simplifying Load Balancing in Web Applications When building a robust web application, ensuring that it can handle multiple users smoothly is crucial. This is where 'load balancing' comes in - it's like having several doors to a shop, so customers don’t have to queue up at just one. In technical terms, we're distributing the user requests across multiple servers. In our journey, we encountered a few hurdles and solved them one by one. Here’s how we did it: 1. The Art of Forwarding Requests: Imagine you’re sending letters (user requests) to a friend (the server). But instead of one friend, you have several who can respond. We created a system where these letters are sent to any one of your friends who's available, ensuring no single friend is overwhelmed. 2. Addressing the Envelopes Correctly: We hit a snag - the letters were being sent, but they had the wrong return address, confusing the recipients. By adjusting the 'return address' on the letters (known as the Host header in web terms), we made sure each letter reached its destination correctly. 3. The Right Stamp for the Letter: Initially, we were putting every possible stamp (headers) on the letters, but not all were needed. So, we carefully selected only the necessary stamps, making sure the letters weren’t overloaded with unnecessary information. 4. Dealing with Misdirection: At one point, our letters were being sent correctly, but they seemed to vanish - like sending them to a house that doesn’t exist (HTTP 404 error). After double-checking the address and making sure our friends were ready to receive the letters, we fixed this issue. 5. Open Lines of Communication: Finally, we made sure that the way we send letters (synchronous vs. asynchronous communication) was efficient. This ensured that our system could handle many letters at once without getting bogged down. In essence, we fine-tuned the process of directing traffic to ensure a smooth experience for the users. It’s like ensuring a well-organized event where guests can move freely without any bottlenecks. This analogy should help non-technical readers understand the complexity behind load balancing in a web application in a more relatable way. Feel free to adjust the language or details to better suit your blog's style and audience!
by Garry Lowther 11 Oct, 2023
Recruiters can benefit from various good business practices in today's competitive job market. Here are some key practices that can help recruitment agencies and professionals succeed: 1. Candidate-Centric Approach : - Personalization : Tailor your communication and job recommendations to individual candidates, showing that you understand their unique skills and career goals. - Transparency : Be honest and transparent about job opportunities, including company culture, compensation, and job requirements. 2. Leverage Technology : - Applicant Tracking System (ATS) : Use an ATS like TriSys to streamline candidate management, automate administrative tasks, and enhance collaboration among team members. - AI and Data Analytics : Implement AI-driven tools for candidate sourcing, screening, and predictive analytics to make data-informed decisions. 3. Build Strong Employer Relationships : - Understanding Client Needs : Take the time to understand your clients' businesses, their cultures, and their specific hiring needs to provide better candidate matches. - Communication : Maintain open and consistent communication with clients to keep them informed about the recruitment process. 4. Effective Networking : - Online and Offline Networking : Build a strong professional network both online (e.g. LinkedIn) and offline (e.g. industry events). Networking can lead to referrals and valuable connections. 5. Diversity and Inclusion : - Diverse Candidate Sourcing : Make an effort to source candidates from diverse backgrounds, promoting inclusion and reducing bias in hiring. - Inclusive Language : Use inclusive language in job postings and communication to attract a broader range of candidates. 6. Candidate Experience : - Responsive Communication : Provide timely feedback and updates to candidates throughout the recruitment process. - Feedback Loop : Seek feedback from candidates about their experience and use it to improve your processes. 7. Continuous Learning : - Stay Informed : Keep up-to-date with industry trends, emerging technologies, and best practices in recruitment. - Professional Development : Invest in ongoing training and development for yourself and your team. 8. Compliance and Ethics : - Data Privacy : Ensure compliance with data protection regulations (e.g. GDPR) and handle candidate data responsibly. - Ethical Practices : Uphold ethical recruitment standards, treating all candidates and clients with fairness and respect. 9. Measurable Metrics : - Key Performance Indicators (KPI's) : Define and track KPIs to assess the effectiveness of your recruitment efforts, such as time-to-fill, quality of hire, and cost-per-hire. 10. Adaptability : - Flexibility : Be adaptable in responding to changes in the job market, economic conditions, and shifts in candidate expectations. 11. Content Marketing : - Content Creation : Share valuable content, such as blog posts, webinars, or whitepapers, to establish expertise in your niche and attract both clients and candidates. 12. Client Education : - Educate Clients : Help clients understand the importance of a streamlined and efficient hiring process, offering guidance on improving their hiring practices. 13. Feedback Loop : - Feedback and Improvement : Regularly gather feedback from both candidates and clients to identify areas for improvement in your processes. Good business practices in recruitment not only enhance the quality of your service but also build trust with both candidates and clients. These practices can contribute to long-term success in the competitive field of recruitment. If you are considering starting a new recruitment agency, we published some good advice a few years ago.
More posts
Share by: