Methods based on automatic visual and audio recognition

Automatic nature observations, or machine observations, are data collected either entirely or partially automatically. Apart from data collection, automation can also aid in species identification, for instance, through machine vision applications using artificial intelligence for image or sound recognition.

On this page:

• Machine observations in nature monitoring
• Automatic image and sound recognition methods in collecting species data
• Data management of machine observations needs to be developed
• References

Machine observations in nature monitoring

Automatic nature observations or machine observations refer to observations that have been collected completely or partially through automated means. The data collection process occurs using technical devices, either fully automatically or with human involvement (such as taking photographs or making audio recordings). In addition to collecting observations, automation can be used for species identification, leveraging machine vision applications based on artificial intelligence (AI) and pattern or sound recognition. Artificial intelligence refers to computer programs capable of performing tasks commonly associated with intelligence. Machine learning, on the other hand, refers to artificial intelligence that learns independently from the data it encounters. Artificial intelligence is a broader term and, for example, a game camera that turns to a moving target can be an example of artificial intelligence that is not machine learning.

Artificial intelligence can be taught with satellite images and other materials to interpret, for example, changes in tree species or water levels in large areas (Jussila et al. 2023). However, remote sensing-based habitat interpretations always require reliable field survey data to guide the interpretation and ensure the quality of the generated information. Remote sensing methods are also planned for use in tasks such as mapping large raptor nests in sparsely populated regions. However, most significant benefits from remote sensing methods for species inventories and monitoring may be related to tracking suitable habitat quantity and quality. More information about remote sensing methods on the interpretation of nature data: Remote sensing.

Automatic sound and image recognition methods in collecting species data

Sound recognition

Automatic observations play an increasingly significant role in collecting species data. Automatic sound recording and automatic recognition of sounds have been put into routine use, for example, in the monitoring of bats implemented by The Finnish Museum of Natural History. At established biological stations, devices placed in fixed locations record bat sounds, which are later analyzed after the field season. In recent years, the use of automatic recorders for monitoring has been expanded through volunteer networks. Additionally, the method of automatic recorders has been tested for surveying and monitoring hibernating bats.

In bird research, automatic sound recognition has been applied in projects such as the LIFEPLAN and in the development of the Muuttolintujen kevät (Spring of Migratory Birds) mobile application. In these projects, citizen science contributions have supported machine learning by enabling volunteers to identify bird species from audio recordings, thereby creating training data for the method.

Furthermore, ongoing research collaborations between Luomus and the University of Jyväskylä involve testing automatic sound recognition methods, comparing them with traditional breeding bird monitoring techniques conducted at the same locations. Audio recording methods have also been tested in determining raptors’ territories, but further development is still needed.

Character recognition

Character recognition is used in nature data collection for many different purposes. For instance, there are several applications available to hobbyists that identify species from user-captured photographs. They rely on artificial intelligence, which compares the details within an image and their relative positions to a large dataset of known species. Often, these apps calculate confidence scores indicating whether the photographed subject belongs to a species already trained in the AI model. The models for automatic species recognition continue to evolve, with active contributions from app users who help identify species from images, thereby enhancing the AI models.

One such freely available application is the iNaturalist, which identifies species from user-submitted photos and audio recordings using AI. In iNaturalist, other users verify the identifications made by the application. Confirmed identifications are stored in the Finnish Biodiversity Information Facility database, making them available for research and expert use. Additionally, Muuttolintujen kevät (Migration Birds Spring) app, assists users in identifying bird species based on their calls, but at the same time these user-contributed audio recordings and associated identifications contribute to scientific research.

Automatic character recognition has also been piloted in several systematic species monitoring efforts. For example, in the monitoring of zooplankton, the automatic classification of plankton organisms has been tested and implemented successfully. An automatic measurement station that takes pictures of the water sample at regular intervals produces tens of millions of pictures per year, so monitoring based on traditional species identification with the same intensity would not be possible. From these same locations, diverse physical and chemical data are collected, enabling the integration and analysis of various information.

The character recognition method has been developed and optimized in cooperation between the Finnish Environment Institute and LUT University (Eerola et al. 2020). Comparative studies with traditional microscopy methods demonstrated the effectiveness of this approach in monitoring cyanobacteria (Kraft et al. 2021). Using the latest techniques, it has been possible to identify up to 98% of the imaged organisms. Currently, images are automatically transferred from monitoring stations to databases, enabling nearly real-time cyanobacteria monitoring. Internationally, significant progress has been made in automatic image recognition methods for nocturnal moths (Bjerge et al. 2021) and in Finland, these methods are being tested alongside traditional monitoring approaches.

Preliminary testing of automatic species recognition has also been conducted in various other monitoring contexts. However, in seal monitoring conducted by the Natural Resources Institute Finland, the challenge lies in the low contrast of seals resting on rocks, making automatic recognition difficult due to the varying rock colors. The use of automatic image recognition and counting from photographs is rapidly advancing, with citizen science playing a role as volunteers identify and count individuals from images. Compared to traditional species monitoring methods, the cost-effectiveness of automatic devices allows for broader and continuous monitoring in many cases.

In addition to identifying species, character recognition can also be used to recognize individual animals. Photo-ID individual identification method has been particularly tested and applied in the case of Saimaa ringed seals (Koivuniemi et al., 2019) and Baltic grey seals (Heiskanen, 2022). However, automating this method has proven challenging, and currently, individual-level monitoring relies heavily on volunteer efforts. Photo-ID individual identification is planned to be used nationally for monitoring other species in the coming years. Beyond estimating population size, individual identification provides valuable insights into population structure. When combined with genetic information, individual identification significantly enhances the reliability of population assessments.

Data management of machine observations needs to be developed

More and more species observations are made based on automatic collection and determination. It is estimated that the volume of such data may soon surpass the number of observations made using traditional methods. Inexpensive devices, open and easy-to-use applications make it possible to collect large amounts of data. The growing amount of data requires new methods for storing, analyzing and sharing data. To ensure data comparability despite technological advancements, careful planning and documentation of methods and data management are essential.

In addition to the development of the different stages of data processing, ensuring the quality of machine observations and quality variation must be taken into account. The quality of machine observations varies depending on the data collection methods and analysis techniques. While AI-based identifications can be more accurate than human determinations, AI systems may introduce systematic biases into the dataset.