Commit 46437f9a authored by Matthew Wilcox's avatar Matthew Wilcox Committed by Linus Torvalds
Browse files

radix-tree: fix race in gang lookup 



If the indirect_ptr bit is set on a slot, that indicates we need to redo
the lookup.  Introduce a new function radix_tree_iter_retry() which
forces the loop to retry the lookup by setting 'slot' to NULL and
turning the iterator back to point at the problematic entry.

This is a pretty rare problem to hit at the moment; the lookup has to
race with a grow of the radix tree from a height of 0.  The consequences
of hitting this race are that gang lookup could return a pointer to a
radix_tree_node instead of a pointer to whatever the user had inserted
in the tree.

Fixes: cebbd29e ("radix-tree: rewrite gang lookup using iterator")
Signed-off-by: default avatarMatthew Wilcox <willy@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Konstantin Khlebnikov <khlebnikov@openvz.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 3c1da7be

include/linux/radix-tree.h

+16 −0
Original line number Diff line number Diff line
@@ -378,6 +378,22 @@ radix_tree_iter_init(struct radix_tree_iter *iter, unsigned long start) 
void **radix_tree_next_chunk(struct radix_tree_root *root,

			     struct radix_tree_iter *iter, unsigned flags);



/**

 * radix_tree_iter_retry - retry this chunk of the iteration

 * @iter:	iterator state

 *

 * If we iterate over a tree protected only by the RCU lock, a race

 * against deletion or creation may result in seeing a slot for which

 * radix_tree_deref_retry() returns true.  If so, call this function

 * and continue the iteration.

 */

static inline __must_check

void **radix_tree_iter_retry(struct radix_tree_iter *iter)

{

	iter->next_index = iter->index;

	return NULL;

}



/**

 * radix_tree_chunk_size - get current chunk size

 *

lib/radix-tree.c

+10 −2
Original line number Diff line number Diff line
@@ -1019,9 +1019,13 @@ radix_tree_gang_lookup(struct radix_tree_root *root, void **results, 
		return 0;



	radix_tree_for_each_slot(slot, root, &iter, first_index) {

		results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));

		results[ret] = rcu_dereference_raw(*slot);

		if (!results[ret])

			continue;

		if (radix_tree_is_indirect_ptr(results[ret])) {

			slot = radix_tree_iter_retry(&iter);

			continue;

		}

		if (++ret == max_items)

			break;

	}
@@ -1098,9 +1102,13 @@ radix_tree_gang_lookup_tag(struct radix_tree_root *root, void **results, 
		return 0;



	radix_tree_for_each_tagged(slot, root, &iter, first_index, tag) {

		results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));

		results[ret] = rcu_dereference_raw(*slot);

		if (!results[ret])

			continue;

		if (radix_tree_is_indirect_ptr(results[ret])) {

			slot = radix_tree_iter_retry(&iter);

			continue;

		}

		if (++ret == max_items)

			break;

	}

CRA Git | Maintained and supported by SUSTech CRA and CCSE