How Do We Predict Customer Behavior in Seconds with Artificial Intelligence?

How Do We Predict Customer Behavior in Seconds with Artificial Intelligence?In the fast-paced world of e-commerce, every company has a magical question on its mind: "Which of my customers will shop again, and when?" The answer to this question can fundamentally change everything from personalized discounts and perfectly timed notifications to inventory management and marketing strategies. So, do we need a crystal ball to answer this question? No. All we need is a smart artificial intelligence that can listen to what the data is whispering.This is precisely where we are pulling back the curtain on our project, which predicts our users' purchase intent in the next 1, 3, 5, 7, and even 10 days with surprising accuracy.Every Click Tells a StoryIt all starts with the digital footprints our users leave behind. Their browsing on the homepage, their a-product inspections, and their eventual purchases... All of these are more than just data points for us; they are behavioral stories.Before training our model, we created a "digital diary" for the past 8 weeks for each of our users. This diary includes not only "what" the user did but also "when" and "how often." But we went deeper and asked some critical questions to better understand the current situation: When was the last time the user visited our site? How many days have passed since their last purchase? How many times did they shop in the last month? What day is it today? Is it the weekend? What time of day? Thanks to this approach, we not only had a film of the past, but also a clear snapshot of the "moment" we would predict the future from.An AI with Memory and the Ability to FocusWe needed a special brain to make sense of these rich "behavioral stories." So, we built one of the most advanced deep learning architectures.1. Powerful Memory (LSTM Layers)At the heart of our model lies a technology called LSTM (Long Short-Term Memory), which mimics the memory ability of the human brain. With this technology, our model can remember a user's 8-week behavioral sequence from start to finish. It understands how a browsing session from last week can trigger a purchase today.2. Attention MechanismBut just remembering is not enough. The important thing is to know which memories are more valuable. This is where our model's "superpower" comes into play: the Attention Mechanism.Think of it this way: When studying for an exam, you don't memorize every word in the book; you underline the parts you think are important. The Attention Mechanism does exactly that! It focuses on the most critical moments that signal a purchase among the hundreds of events in the user's 8-week history and concentrates its attention there. Perhaps a user browsing the same category for three consecutive days is a much more important signal than a purchase from a week ago. Our model can discover this on its own.3. The Master Chef Who Finds the Best Recipe (KerasTuner)To ensure we found the best model, we left nothing to chance. We used a "master chef" named KerasTuner, which automatically tries hundreds of different model architectures and settings. This system worked tirelessly for us until it found the most delicious, or in this case, the most accurate "recipe," and presented us with the most performant model possible.How Do We Make Sense of the Results?Our model doesn't give us a simple "will buy" or "will not buy" answer. It offers something much more valuable: probability scores. For example, "User A has an 87% chance of making a purchase in the next 3 days."This gives us incredible flexibility. For instance: High-Value Campaigns: We can use our marketing budget most efficiently by sending a special discount coupon only to users the model is 90% or more confident about. General Announcements: We can make sure we don't miss anyone by announcing a new product to a wider audience that the model gives a 60% or higher probability. So, by adjusting the model's "confidence level" according to our business goal, we can strike a perfect balance between accuracy and the reach of our target audience.A Data-Driven FutureThis project is an exciting testament to what's possible when we combine big data and modern artificial intelligence. We no longer rely on fortune tellers to predict our customers' next move but on the power of our own data and the algorithms that interpret it intelligently. This is just the beginning; our future goal is to strengthen the bond between us and each user by offering them a personalized experience exactly when they need it.

How Do You Dynamically Pass Your CRM Audiences to Advertising Channels?

How to Dynamically Sync CRM Audiences to Ad ChannelsSay goodbye to static lists and hello to living audiences.The golden rule of digital marketing—reaching the right audience at the right time—is often missed due to inefficient batch processing methods. The key to effective, real-time targeting through CRM audience synchronization lies in a modern approach: event-driven architecture. This method captures real-time events using Pub/Sub, processes them with technologies like BigQuery streaming, and instantly sends your valuable customer data to Google's Customer Match and Meta's Custom Audiences. This dynamic system not only accelerates data transfer but also intelligently manages rate limits to ensure seamless communication, avoiding the API walls of advertising platforms.Think about it for a moment. A customer favorites a product in your app. Instead of waiting for the next day's manual CSV upload or a 15-minute delay from Zapier, you could instantly add this person to your Meta or Google audience and serve them a relevant ad. By the time those manual processes run, the "moment" is long gone.In this article, we'll break down how we overcame the time costs, flexibility constraints, and—most importantly—the "rate limit" walls imposed by third-party tools like Zapier. Let's explore how to transform your CRM data from a static list into a living, breathing organism that's constantly updated for your ad channels.Dreams vs. Reality: Where Tools Like Zapier Fall ShortWith the promise of automation, tools like Zapier and Make (formerly Integromat) are excellent starting points for non-coders. "When a user is added to the CRM, add them to a Facebook Custom Audience." It sounds simple, right? But as your business scales and data velocity increases, this magic wand quickly becomes a set of shackles.1. The Time-Cost Trap: How "Instant" Is It, Really?Zapier and similar platforms run on polling triggers, checking for new data at set intervals. Depending on your plan, this could be every 5, 10, or 15 minutes. In a world where purchase decisions are made in seconds, 15 minutes is an eternity. You miss the critical "hot moment." Imagine the operational drag and cost of syncing thousands of users one by one through these delayed cycles.2. The Rate Limit Wall: Hitting the Brakes When You Need to AccelerateEvery API has a carrying capacity: its rate limit. This is the number of requests you can make in a given time frame. What happens during a major campaign when you need to add tens of thousands of users to an audience at once? Zapier will collide with the rate limits of both its own platform and the ad channel (Meta, Google, etc.). Just when you need to floor it, the system forces you to slow down, leaving you powerless.3. The Flexibility Nightmare: Trapped Inside the BoxThese tools provide pre-built templates. But what if you need a complex segment like, "users who have added items to their cart three times in the last 30 days but haven't purchased, and who also opened our most recent email"? What if you need to enrich, clean, or merge data from multiple sources before sending it? This is where you hit the rigid flexibility ceiling of off-the-shelf solutions.The New Game Plan: Building Our Own Highway with APIsRecognizing these bottlenecks, we made a decision: instead of getting stranded in someone else's rental car, we would build our own highway. This highway is built on direct API integration.Here are the steps of our robust solution:Step 1: The Heart of the Data – Google BigQueryIn compliance with security and privacy standards, all our user data is hashed (e.g., PII like emails and phone numbers are encrypted) and streams dynamically into Google BigQuery. Every time a user takes an action—opens the app, views a product, adds to cart—that event data is processed and lands in our massive data warehouse almost instantly. This is our single source of truth.Step 2: Polishing the Raw Diamond – Data Quality and ModelingThe data flowing into BigQuery is raw. Our job is to turn that raw diamond into a brilliant-cut gem. In our proprietary data processing layer, we:Clean the Data: We scrub for incorrect, incomplete, or inconsistent entries.Enrich the Data: We combine behavioral data with demographic or historical purchase data to build a 360-degree customer profileCreate Dynamic Segments: Using SQL queries, we can generate complex, real-time segments in seconds, such as "VIP customers who have favorited more than 3 products in the last 24 hours."This stage is the brain of our advertising campaigns. It's where we decide who to target, when, and with what message.Step 3: Light-Speed Delivery – API Push to Ad ChannelsNow we have clean, enriched, and perfectly modeled audiences ready for activation. This is where the magic happens:Our custom application takes these segments and pushes them directly to the Google Ads, Meta (Facebook/Instagram) Ads, and TikTok Ads APIs.What does this give us?Zero Latency: The moment data lands in BigQuery and is processed, it is added to the target audience on the ad platform within seconds. A customer could see a personalized ad on social media before they even close your app.Limitless Scalability: Rate limits are no longer a concern. With our own code, we can intelligently manage API requests, queue them, and use batch processing to send tens of thousands of users in a single push without hitting API limits. We are in full control.Maximum Flexibility: We can create any segment imaginable and instantly activate it. We can build and leverage niche, high-value audiences with our own business intelligence—audiences far too complex to create within the ad platforms' native UIs.So, How Does It Actually Work?Achieving these three superpowers—speed, scalability, and flexibility—is the result of the right engineering and architectural choices. Let's take a closer look at the technical solutions.1. How We Achieve "Zero Latency": Event-Driven ArchitectureUnlike Zapier's periodic polling mechanism, our system is event-driven.Real-time Data Streaming: When a user takes an action, that event is captured and streamed to BigQuery instantly. We don't wait for the next sync cycle.Automatic Triggers: As soon as new data arrives in BigQuery, it automatically triggers the next step. A serverless function, like Google Cloud Functions, picks up this new data and immediately initiates the data processing pipeline. In short, one domino instantly knocks over the next. There is no waiting, only flow.2. How "Limitless Scalability" Is Possible: Smart Queue Management and BatchingTo avoid crashing into the rate limit wall, we manage API requests intelligently.Message Queues: After data is processed, instead of sending each user directly to the ad platform's API, we push them into a message queue like Google Cloud Pub/Sub. This acts as a buffer, absorbing sudden spikes in load and allowing the system to breathe.Controlled Workers: "Worker" services listen to this queue and process the entries at a controlled rate (e.g., "send no more than X requests per minute"), respecting the API limits of each ad platform. This way, whether we have 100 new users or 100,000, the system never clogs; it simply works through the queue.Batching: Instead of sending 1,000 individual API requests, we group users into batches—as permitted by the API—and send them in a single request. This dramatically improves API efficiency.3. Where "Maximum Flexibility" Comes From: The Power of SQL and Custom CodeOur flexibility stems from our freedom to work directly with data and code, unconstrained by a third-party tool's interface.Centralized Logic with SQL: All of our segmentation logic lives in powerful SQL queries running on BigQuery. When the marketing team dreams up a highly complex audience, we can build it with a few lines of SQL instead of fumbling through a series of dropdowns in a Zapier workflow. We own the rule set completely.Custom Data Transformation: Our data processing layer is built with our own code, written in languages like Python. This allows us to enrich data however we see fit, merge it with various sources, and perfectly format it for each ad platform's specific requirements. The only limit is our imagination, not the technology.These strategies allow us to move from a reactive, constrained world to a proactive, fully controlled playing field.Conclusion: From Static Lists to Living OrganismsZapier and similar tools are like a public bus—they'll get you from point A to point B. But if there's traffic, you'll be stuck waiting, and you can't deviate from the route. Building your own API integration, on the other hand, is like chartering a private jet: it's fast, flexible, limitless, and entirely under your control.Stop treating your CRM data as "dead" lists that are updated weekly or daily. They are living organisms reflecting the real-time intent and behavior of your customers. When you feed these living organisms in real time via APIs, your ad campaigns also become living, breathing systems that perform better than ever before.

Back to School and Return to City Life: 2025 Trends and Consumer Behaviors

Back to School and Return to City Life: 2025 Trends and Consumer BehaviorsThe back-to-school period signifies a new beginning not just for students and their parents, but for the entire city. This process combines the concepts of "Back to School" and "Return to the City," presenting a major opportunity for the retail and marketing world. Understanding the trends and consumer behaviors of this period in 2025 is critically important for brands to define their strategies correctly. As families focus on both traditional school needs and the new dynamics of urban life in their preparation for the new term, technological advancements and AI-powered platforms are shaping this shopping journey. In this article, we will conduct a detailed analysis of the general market outlook, target audience, and digital marketing trends of this period.Market Overview of the Back to School & Return to the City PeriodThe back-to-school period is a high-stakes process for families, with the average family spending 10,000 TL in 2024. The largest portions of this expenditure were on stationery, clothing and shoes, and books. Parents are inclined to spend more with encouragement from their children; according to a 2024 survey, 61% of parents stated that their children encouraged them to spend more during the back-to-school season. This indicates that children play a significant role in purchasing decisions and that brands need to shape their target audience strategies towards both parents and children. Furthermore, online shopping has gained increasing momentum during this period, a trend that has strengthened over the years. This highlights the importance for brands to have a digital presence and run effective campaigns.Across the market, consumer spending habits and expectations are evolving with increased competition and technological advancements. While global retail sales are expected to grow by 4% in 2025, e-commerce is projected to account for 21% of retail sales. This requires brands to establish a strong presence in both physical and digital channels. Consumers, especially during peak periods like "Back to School" and "Return to the City," want to discover products in more natural and intelligent ways. In this context, visual search tools like Google Lens and other AI-powered technologies are coming to the forefront by personalizing and simplifying the shopping experience. These trends demonstrate the necessity for brands not only to showcase their products but also to offer a value-oriented, competitive, and technology-supported shopping experience to consumers.Target Audience Analysis and Consumer BehaviorsThe target audience during this period encompasses a wide spectrum, including both parents and students. In Turkey, 51% of X (formerly Twitter) users are parents with children under the age of 18. Of these parents, 28% have at least two children. This data shows that platforms like X hold great potential for reaching and engaging with parents. While parents research their children's school needs, students use different platforms to discover products that reflect their personal style and school life. Therefore, brands need to create original and engaging content that appeals to both audiences. For example, on Pinterest, both students and parents conduct shopping-related searches using keywords like "school supplies," "backpack organization," and "back-to-school checklist."Consumer behaviors are shaped by their interactions, especially on digital platforms. On TikTok, users generated 9.6 billion video views for hashtags related to "Back to School" (#B2S). The content of these videos generally focuses on communities, shopping, schools, and preparation. Among the most popular hashtags are #okul (school) (770M views), #üniversite (university) (466M views), and #okuladönüş (back to school) (261M views). This shows that brands can reach large audiences through popular hashtags and viral content. Additionally, consumers tend to set aside their brand loyalty during this period and become more open to new brands. According to Google's internal data, 51% of consumers indicated they were open to new brands during last year's peak preparation season. This demonstrates that brands need to develop effective strategies to acquire new customers in a highly competitive environment.Category and Digital Marketing TrendsDigital marketing trends are focusing on the consumer's search and discovery processes. The rise of online shopping, particularly during this period, makes it mandatory for brands to have a strong digital presence. Platforms like Pinterest play a significant role in converting users from "inspiration seekers" to "ready-to-shop" consumers. Student Pinners are 50% more likely than non-students to save shoppable Pins during the school season. Furthermore, categories such as fashion, home decor, art, beauty, and DIY & crafts are among the most shopped on Pinterest. This data emphasizes the importance for brands to showcase their products on visually-oriented platforms and create shoppable content.The article should also address digital marketing strategies. AI-powered tools enable brands to run more personalized and efficient campaigns. For example, tools like Google's AI Max help capture demand through search term matching and auto-generated content. Similarly, campaigns like Demand Gen can be used to drive more conversions on YouTube and other Google networks. On TikTok, users shape their purchasing decisions by interacting with videos on topics like fashion, hairstyles, and meal prep. These platforms allow brands to connect with their target audience and promote their products in a natural, fun, and informative way. Consequently, campaigns prepared for the "Back to School" and "Return to the City" period can reach wider audiences, holding the potential to increase sales.Back-to-School Shopping Trends (fashion, beauty, home life, food & beverage)Back-to-school shopping is a time when students and parents focus not only on school supplies but also on their personal style, living spaces, and daily routines.In the fashion category, students prefer comfortable and casual clothing, while parents search for clothes for special occasions like school trips for their children, as well as for themselves, with styles like "cool mom" outfits. Bags are also a major focus; searches for "university bag," "college bag," and "cute school bags" show a significant increase.In the beauty realm, practical solutions like "easy hairstyles" and "simple makeup looks" are prominent for time-crunched students, while parents also search for "school hairstyles" for their children.Distinct trends are also observed in the home life and food & beverage categories. Students look for decoration ideas to make their dorm rooms feel "like home," searching with keywords like "cozy dorm room" and "study desk decor." Parents, on the other hand, focus on reorganizing the home to ensure their children are ready for the new term. They seek inspiration for study areas and storage solutions with searches like "kids' study desk ideas" and "school bag storage."In the food & beverage category, both students and parents research ways to prepare practical and nutritious meals for busy school days. Searches like "easy healthy meal prep," "school lunch ideas," and "healthy breakfast for kids" are among the most popular topics of this period.Frequently Asked QuestionsWhen does the back-to-school period start and how long does it last?The back-to-school period generally begins about a month before schools open, in August, and continues until the end of September. However, conversations and searches related to shopping and preparations start to increase as early as July. It is recommended that brands launch their campaigns in August and continue them throughout September to make the most of this busy period.What are the most preferred products for back-to-school shopping?The categories with the highest spending during back-to-school shopping are stationery, clothing and shoes, and books. In addition, technological products, school bags, lunch boxes, and home decor items also attract significant interest. Searches like "backpack organization," "cute school supplies," and "dorm room ideas" are particularly popular on digital platforms.

Hello

Sustainability in E-Commerce

Sustainability is not a factor that can be ignored in e-commerce—it must not be. The rising trend of environmental sustainability is humanity’s reflex against the destructive “grow or die” imperative of capitalism. In e-commerce, sustainability can span from business models to packaging materials, and with younger generations increasingly taking the lead, it will become ever more important in the years ahead.Awareness of environmental issues and sustainability has long been a trend in global e-commerce. To successfully meet growing consumer demands for sustainability, online retailers have been focusing on topics such as corporate social responsibility and eco-friendly approaches within their supply chains.In this article, we will cover: What Is Sustainability? Sustainability in the E-Commerce Sector How the New Generation Views Sustainability Example Brands What Is Sustainability? (Revisited) “Sustainability,” a concept we’ve been hearing about increasingly in recent years, aims—as its name suggests—to ensure that all living beings around the world can continue to exist indefinitely. The ultimate goal is to build a world where we do not compromise today’s standard of living, where future generations can equally benefit, where resources are used in a controlled manner, and where everyone can live healthily and happily. Every effort—including those in e-commerce—must be in harmony with nature. No individual or corporate action should disrupt ecological balance; rather, it should directly or indirectly contribute to its long-term stability. A. Getting Started in E-CommerceE-commerce has spread rapidly worldwide since the 1990s. In Turkey, it has become especially popular over the last 20 years. Naturally, with this boom in online shopping comes an increase in waste materials. Shipping boxes, plastic bags, branded plastic wraps from courier companies, and so on accumulate as large piles of recyclable or non-recyclable waste. In this situation, traditional businesses moving into e-commerce have begun to focus more on eco-friendly practices and must become increasingly cautious about sustainability.B. Sustainability Around the WorldScientists and international bodies have made broad calls and warnings urging humanity to protect nature and ensure sustainable living on Earth. These calls demand fundamental changes and urgent measures to avoid further environmental harm. Ensuring and supporting sustainable living has become inevitable for nations. Therefore, while maintaining economic development, countries must avoid actions that harm the environment and develop strategic plans for sustainable development. The urgency of sustainable development grows daily. Many international organizations are actively working on sustainability. Bodies such as the European Union (EU) and the United Nations (UN) have mobilized to promote sustainable growth, devising various strategies and policies. Under its Europe 2020 strategy, the European Commission defined three main growth priorities, one of which is “promoting a more resource-efficient, greener, and more competitive economy.” Meanwhile, the United Nations took an even more comprehensive step. In 2000, the UN set the Millennium Development Goals (MDGs) and in 2015 expanded them into the Sustainable Development Goals (SDGs). The SDGs are considered the most comprehensive action plan for nations to achieve sustainable development. These goals are financially supported by the World Bank Group (2017) and the International Monetary Fund (IMF, 2020). In September 2015, the UN adopted 17 SDGs as core components of the 2030 Agenda for Sustainable Development—extended versions of the MDGs that concluded at the end of 2015. All UN member states committed to building a sustainable future encompassing the three dimensions of development: environmental, social, and economic.C. Reducing Carbon FootprintLogistics—especially transportation by truck or plane—relies heavily on fossil fuels, increasing carbon emissions. Reducing carbon footprints helps slow and mitigate global warming. In Turkey, a prime example of a company leading carbon footprint reduction is Yemek Sepeti. Handling millions of orders daily, they removed single-use plastic cutlery, drastically cutting plastic waste. Another notable example is Koton, which redesigned its cardboard packaging for multiple uses, thereby reducing its carbon footprint. LC Waikiki launched a project to transform its shipping boxes into toys for children, further promoting sustainability.On their website, LC Waikiki describes the initiative:“LC Waikiki is preparing to win children’s hearts with its new project. According to a LinkedIn post by E-commerce General Manager Ömer Barbaros Yiş, the brand’s shipping boxes will now become toys.”This initiative, while still under development, hints at a creative approach to reducing waste and enhancing customer engagement.D. The New GenerationThe 1970s hippie movement introduced environmental respect and sustainability topics to the wider public. In the 1990s, as e-commerce spread globally, discussions of sustainability in online retail began in earnest. Millennials—born in the ’80s and ’90s—are generally more aware and conscientious than earlier generations (Baby Boomers and Generation X). Economic crises and environmental disasters (e.g., Amazon deforestation, melting glaciers, wildfires in Australia and elsewhere) have made Generation Z even more attentive to sustainability. Generation Z, born into the internet era, often shops online as a matter of course—and sometimes even more than offline. For them, sustainability in e-commerce is critically important. Knowing this, both legacy retailers and pure-play e-commerce startups are embracing sustainability practices as both a business strategy and a demonstration of respect for the environment.E. Sustainable E-Commerce in TurkeyE-commerce has existed in Turkey for over 20 years, but initial distrust of the internet slowed its growth. Over the past five years, sales have surged, especially during the COVID-19 pandemic, mirroring global trends. Because e-commerce developed later in Turkey than in many other countries, both traditional retailers moving online and new e-commerce ventures are rapidly adapting and innovating to meet consumer expectations.Example BrandsSome Turkish companies leading the way in sustainable e-commerce include:E-BebekE-Bebek is often the first name that comes to mind regarding sustainability in Turkey. According to an Instagram post by Sertaç Doğanay, they introduced “detergent refill centers” in major stores. Customers bring their own containers to refill detergent at prices lower than retail, reducing single-use plastic waste and embracing an eco-friendly commerce model. MaviMavi took a step by switching from disposable plastic shipping bags to resealable, reusable pouches, minimizing plastic usage. They also launched the “Mavi Transformation” project, turning 593,750 recycled PET bottles into high-quality denim through their expertise. CEO Cüneyt Yavuz stated: “Using innovation and technology, we’re transforming denim and aim for 100% sustainable All Blue products by 2030.”LC WaikikiLC Waikiki’s “Green Collection” of eco-friendly, organic apparel demonstrates its commitment to sustainability. They also announced plans to convert shipping boxes into toys, as hinted by E-commerce GM Ömer Barbaros Yiş on LinkedIn.Boyner GroupBoyner Group highlights its sustainability and CSR efforts under four pillars: Social Impact, Environmental Impact, Workplace Democracy, and Our Value Chain. They publish transparent sustainability reports from 2008 to 2020, showcasing their eco-friendly initiatives and commitment to a sustainable future.If you found this blog post on sustainable e-commerce helpful, please share it on social media so others can benefit!

Adapting New Tool for Our Automated Jobs: Apache Airflow

Performance marketing combines advertising and innovation to assist merchants and affiliates expand their companies in any aspect. Each retailer's campaign is carefully targeted, ensuring that everyone has a chance to succeed and win. When all the operations on each side are done correctly, performance marketing offers win-win situations for both merchants and affiliates. As Tech Team, we gave a decision about writing new blogs about the digital marketing projects we build with the software engineering skill sets. We mainly produce new solutions for different brands and our main goal is to increase their performance with data analysis and automation projects. On that sense, we decided to publish new blogs related to our projects. What do we produce as a team? As a tech team, we adopted the modern performance marketing ideology and as a result of this, we produce an automation project using Airflow to schedule product reports for our customers. On our product reports, we are gathering information about the unique codes, availability, discounted prices, discount percentages and many more related features about the products by visiting their URLs. Moreover, we have a solid background on Google Sheets to add more power on our automation projects for more monitoring & reporting purposes. Furthermore, we also used docker container technology to adapt the Airflow environment into a virtual environment for the problems that we can face during deployment phase. To be more clear, Airflow is an automation tool to create data pipelines for multiple purposes. The main reason we work with Airflow is, rather than cron jobs, Airflow provides us a UI service to monitor all the processes in almost real-time. In addition to that, analyzing the logs on the platform has a significant impact on catching and regulating errors during the process. Moreover, when an error occurs on the system during the processes, with the configuration file, we can get emails for any kind of errors and it provides us a service for instant interference. First of all, the whole process depends on scraping and on the performance side of the whole process, we mainly use parallel threads working asynchronously and with that way we can scrape all the data in minutes for a large scale of URLs to be checked. We designed multiple virtual machine templates on Google Cloud which can operate all the required tasks in a given order. The most significant part of the order is to deliver data through the processes in a single template for the upcoming reports on the Airflow platform. What is Airflow? Let’s get more deep inside into the main structure of Airflow to understand how it works and how we can adapt this data pipeline environment to different projects for future purposes. “A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more”[1]. Basically we can think of Airflow as a way more complex version of cron jobs. In Airflow we are using workers as threads to operate all the Tasks. First of all, we need to know that all Airflow tasks work on a pre-structured object named ‘DAG’, which is a terminology to annotate each task we scheduled on the Airflow system. For example; with DAG( "Company_X_Product_Report", schedule_interval='@daily', catchup=False, default_args=default_arguments ) as dag: 1- We can define DAGs using Context Manager. DAG structure has so many features such as time scheduling, naming and retry options in case of any errors, helping us to regulate and set each of them on the data pipeline environment. In addition to that, we also have default-arguments to add more features on the DAG structure. For example; default_arguments = { 'owner':'AnalyticaHouse', 'start_date':days_ago(1), "sla": timedelta(hours=1), 'email': ['analyticahouse@analyticahouse.com'], 'email_on_failure': True, } 2- We used multiple functions to operate each task in a DAG. To be more clear, whole process can be expressed as: url_task >> scrape_task >> write_to_sheet_task >> find_path_task >> parse_message_task >> write_message_task 3- We also separate each task with corresponding functions using PythonOperators. Mainly these operators are responsible for the Python functions handling the required duties. One of the most used ones are Bash Operators and Python Operators for executing bash and python scripts. In our project, we mainly used Python Operators for executing each function one by one. In this example we showed that each scheduled python function needs to get implemented in such a fashion that Airflow will interpret all the implemented format as a job. We used more than 5 main functions to operate the whole scheduled job and we can think like we are adding the scheduled jobs to a stack structure. In this manner, all the PythonOperators will be added into a line and get executed one by one in a row. If one of the tasks fails (if exception raises), all operations fail and Airflow informs us. url_task = PythonOperator( task_id='get_url_data', python_callable=getUrlData, ) Combining the Airflow architecture with Docker Docker is a free and open platform for building, delivering, and operating apps. Docker allows you to decouple your apps from your infrastructure, allowing you to swiftly release software. In that sense, we also used Docker in our development stages and at the end of the day, we created a docker image based on Apache’s docker image on DockerHub[3]. We came up with small changes such as network bridging and port directions for other purposes. Basically, while we are developing the Airflow environment, we are faced with different minor and major problems including packet incompatibilities and version problems. Moreover, before we started working with Airflow, we did research about how we can adapt Docker technology to the Airflow project and then we made several meetings to have a common sense about the advantages that Docker can bring into action. Shortly after, during the development process we got stuck in so many points and every time we crash into the wall, rather than removing Airflow from the server, we deleted the container that we created and re-deployed it with the same Docker image. In addition to that, composing or creating a Docker container helped us to save so much time during the development part of the project. Advantages: Time saving in case of any unexpected errors during the development phase Easy to configure networking and volume features for the development of the project Isolated environment which is good for managing small bugs and errors Containerized structure brings easy configurations for each build time Challenges during the development Before we start implementing the project, we made a lot of brainstorming to figure out the critical parts of it because every time we come up with an architectural design, we were always adding more features on top of it to make it the best in the industry. We are a team sharing the ideology of creating a product in an impeccable way. As a result of this ideology, the starting point of the project differs from the end product. First, we built the project on Google Cloud VM and then with the other projects we developed, it started to get harder for us to manage and deploy all of them. Then, as a team, we decided to work with Docker to make everything easier. We are a young team eager to learn and build new projects with the latest and reliable technologies. In that manner, we also implemented error-catching modules because while we were developing the project, we faced server and Airflow crashes for no reason. Catching errors has an essential impact on us because customers that we are working with need to receive product reports daily, and during the flow of the whole automation project, in case of any errors, it results in undelivered reports. References [1],[2]-https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html [3] https://hub.docker.com/r/apache/airflow